Microsoft Mission Critical Blog articles

Building an AI Azure Ops Workbench — Part 3: Reverse engineer Azure Architecture Diagrams

zmustafa — Thu, 30 Jul 2026 21:27:01 GMT

In Part 1 we made the case for an operations copilot that runs inside your own tenant. In Part 2, the Azure Support Agent's war room of specialist agents root-caused a live 5xx in about ninety seconds. But every one of those specialists had to be pointed at something first — and defining that "something" is the problem this post solves. Welcome to the second pillar: Map.

Here's a question that sounds trivial and almost never is: which resources make up your application?

Azure knows. It exposes the answer generously, too — Azure Resource Graph will return every resource in the tenant, with full properties, from a single KQL query. The Portal shows every relationship if you click far enough down. Azure Monitor holds the metrics, Microsoft Entra ID holds the identities. Very little about your estate is genuinely hidden.

What's scarce isn't data. It's time. Nobody has a spare afternoon each week to re-derive which forty of your eleven hundred resources constitute "the checkout app," redraw the diagram, and write down why the private endpoint is there. So the map drifts out of date — not because the information went missing, but because keeping it current is work that never wins against a sprint. That's a bandwidth problem, and bandwidth problems are exactly the kind of thing you can hand to an agent that never gets bored.

So the Azure Support Agent builds the map for you and keeps it alive. Five pieces, each feeding the next: Workloads that mean something, a live architecture diagram reverse-engineered from what's actually deployed, an Architecture Memory that remembers design intent, a Know-Me doc your on-call can read at 3 a.m., and an Estate Graph that renders the whole tenant as one navigable picture.

One promise up front, because it shapes every design decision below: mapping your estate never changes it. Every feature in this post is read-only against Azure.

Part one — Workloads

A unit of scope that actually means something

A subscription is an accounting boundary. A resource group is a lifecycle boundary. Neither one reliably equals "an application." So the workbench introduces one that does: the Workload.

A workload is a named set of scopes, and it can mix all four levels Azure gives you — a management group, a subscription, a resource group, or an individual resource ID. Real estates are untidy, so membership is subtractive as well as additive: every scope carries an excludes list, which lets you express "this entire resource group except that one legacy VM" without hand-listing thirty-nine resource IDs and re-listing them next month.

Alongside its scopes, a workload carries the metadata that everything downstream reasons about: workload type, environment, business criticality, data classification and free-form tags. That metadata isn't decoration — the fleet views pivot on it, and the assessments in Part 4 weight their findings by it.

This layer matters more than it first appears, because the workload is the unit of scope for the entire product. Every assessment, every coverage report, every architecture diagram in this post, and every deep investigation from Part 2 is scoped to one. Get this right and everything downstream inherits it. Get it wrong and you're running beautiful analysis on the wrong set of resources.

Autopilot: let the AI propose the boundaries

Defining workloads by hand is perfectly fine for five of them. It does not survive eleven hundred resources, and it really doesn't survive an estate somebody else built.

Autopilot reads the estate and proposes the groupings, reasoning over exactly the signals an experienced engineer would use: tag values, naming conventions, resource-group boundaries, and which resources actually sit together. It's built for real tenants rather than demos — a discovery pass handles up to 5,000 resources (and tells you when it truncated rather than silently sampling), feeding the model in batches of 500 in a map-reduce pattern so a large estate never has to fit inside one context window.

The important part is what happens next: nothing is saved behind your back. Autopilot streams out candidate workloads, each with its reasoning, a confidence level, and a breakdown of what's inside it — and then it stops. Those candidates are proposals until you explicitly save them. You can rename, merge, split, or bin them. This is the same posture as the Launch gate in Part 2: the AI proposes, a human disposes.

Each Autopilot-born workload also remembers its origin — the management group or subscription it was discovered from — which powers a one-click Refresh when the estate moves on, instead of making you re-run discovery from scratch and reconcile by hand.

Groups, for when one app is really five

Most real applications exist several times over: CRM PROD, CRM DEV, CRM DR. Treating those as three unrelated workloads is technically correct and practically useless.

Groups are a non-destructive association on top of workloads — members keep their own identity, their own scopes and their own health score, but you can see them as one family. And because the naming pattern is so consistent in practice, the workbench will suggest the groupings for you: the suggestion pass strips environment tokens (prod, dev, stg, test, dr, shared, sandbox) from workload names and groups whatever stems match. CRM PROD and CRM DEV both stem to CRM, so they're offered as a group.

It's a small feature that does something disproportionately useful: it lets you ask "is this application healthy?" rather than "is this one environment of it healthy?"

A health score you're allowed to argue with

Each workload the Azure Support Agent tracks carries a composite health score out of 100. The interesting part isn't the number — it's that you can take the number apart.

The score is a weighted average of seven signals, and they are deliberately not weighted equally:

Signal	Weight	Why
Monitoring	1.0	Are the right alerts in place?
Telemetry	1.0	Are logs and metrics actually being collected?
Backup / DR	1.5	An unrecoverable workload is the worst failure mode on the list
Performance	1.0	Is anything saturating or trending at its limits?
Ownership	1.0	Does a human own this?
Policy	1.0	Is it governed and compliant?
Tags	0.5	Untagged is untidy, not unsafe

Bands are plain: 80 and above is good, 50 and above is a warning, below 50 is poor, and a workload with no analyzed signals at all reads unknown rather than zero.

Three design decisions are worth pulling out, because they're what make the number usable rather than decorative.

First, it explains itself. The API doesn't return a bare score; it returns the contributing signals, the missing ones, and the weights actually applied. "Why is this 62?" has a real answer you can put on a slide, not a shrug.

Second, unmeasured never masquerades as bad. Signals you haven't analyzed yet are excluded from the denominator entirely — a workload is never punished for a backup assessment nobody has run yet. It's marked unknown and the score is computed from what's actually known. A score that quietly conflates "bad" with "not yet measured" is a score people learn, correctly, to ignore.

Third, you can disagree. The weights are an admin setting. If your organisation's risk appetite says tags matter more than that, change it — the score is a tool for your judgement, not a replacement for it.

The board, the table and the cockpit

Fleet views tend to fail in one of two directions: a wall of rows nobody reads, or a dashboard so abstract it says nothing. The workbench offers three deliberately different lenses.

The table is the working view — one row per workload with environment pills, the score badge, the individual health signals broken out, resource counts and a risk column that folds in retirements and critical findings. It's what you scan on a Monday.

The board is a Kanban-style swimlane view grouped by environment — production, staging, development, test, DR, shared, unknown — which is the fastest way to spot the thing everyone dreads: something sitting in production with a health score in the reds.

The cockpit is the strip across the top: health distribution across the fleet, a composition treemap, an environment × criticality matrix, and a risk ticker. Four small pictures, each answering "where should I be looking?"

Underneath the visuals sits a shared resource taxonomy — every Azure resource type is classified into a category (compute, web, containers, data, storage, networking, security, integration, AI/ML, analytics, monitoring, management) and an architectural layer (edge, presentation, application, integration, data, networking, security, monitoring, shared). That single classification is what lets a donut chart, a treemap and an architecture diagram all speak the same language about the same estate.

And the charts themselves are honest little things: a composition donut by category, a seven-axis health radar (one axis per signal, so a lopsided workload looks lopsided), a sparkline of score trend, and a treemap of resource categories by weight. Click into a single workload and you get an Overview tab — at-a-glance, health and coverage, watchers, security, lifecycle — plus a full resource table, with chips that jump you straight to that workload's architecture, its retirement radar, its policy view.

Part two — The Azure Support Agent's Architecture Designer

Now draw it, from what's actually deployed

This is the part that tends to make people sit up in a demo, so it's worth going through properly.

Point the workbench at a workload, press Generate, and it reverse-engineers an interactive architecture diagram from live Azure data. What follows isn't one big AI call — it's a pipeline with a deliberate division of labour: Azure supplies the facts, the model supplies the reading of them, and code enforces the truth. That split is the whole reason the output is worth trusting.

Step one: read the estate, twice

The workbench resolves the workload's scopes into a set of subscriptions, resource groups and resource IDs, then queries Azure Resource Graph in two passes.

The first pass is deliberately cheap — a lightweight inventory projecting only what it needs to identify things: id, name, type, kind, location, resourceGroup, subscriptionId, sku, identity, zones and tags, plus the resource groups themselves.

The second pass goes back for the interesting part: properties, fetched in chunks of 30 resources at a time. Properties are where the actual topology hides — but they're also enormous and mostly irrelevant, so they're filtered before they ever reach the model. The filter keeps keys that carry relationships and drops the rest:

networkProfile · ipConfigurations · subnet · privateLinkServiceConnections · serverFarmId · storageProfile · agentPoolProfiles · vnetSubnetID · backendPools · routingRules · keyVaultProperties · connectionStrings · siteConfig · hostNames · primaryEndpoints

Two budgets bound the whole thing: roughly 6,000 characters per resource against a 120,000-character total. When the budget runs out, the remaining resources are explicitly marked as omitted rather than silently dropped — the pipeline would rather tell you it ran out of room than quietly draw you an incomplete picture. A sprawling workload degrades gracefully instead of blowing the context window, which is the difference between a feature that works on your estate and one that only works in a screenshot.

Step two: what the model is actually asked to do

The model receives the workload name, the resource count, and a compact JSON array of resources with their trimmed properties. The instruction that matters most is this one: infer relationships from the properties, not from what you know about Azure in general. A model asked "how do App Services usually connect to SQL?" will confabulate a plausible architecture. A model asked "here is serverFarmId and here is privateLinkServiceConnections[], what do they tell you?" is doing evidence-based reading.

It's asked to return a strict JSON contract — a name, a description, nodes[], edges[], groups[], a short rationale, and a confidence value. Nodes carry an arm_id tying them to a real resource. It's permitted at most a couple of conceptual nodes without an ARM ID — "Users", "Internet" — because a diagram with no origin for the traffic is harder to read, and that's a bounded, honest exception rather than a licence to invent.

The call is given generous headroom (16,000 tokens) so a large estate doesn't produce a diagram truncated mid-JSON, and if the model returns nothing it retries once before reporting failure rather than showing you an empty canvas.

Step three: the validation pass, where trust is actually won

This is the least glamorous stage and the most important one. Everything the model returns is treated as a proposal and put through normalization before it's allowed near your screen:

Groups are de-duplicated, and their kind is clamped to the permitted set — subscription, resource group, VNet, tier, custom.
Nodes are de-duplicated, and each arm_id is cross-referenced against the real inventory. Category and layer are re-derived from the actual ARM type rather than trusted from the model. Node metadata is capped at six key–value pairs with length limits, so nobody can stuff an essay into a diagram label. Group references that point at a group that doesn't exist are cleared.
Edges get the strictest treatment: any edge pointing at a node that doesn't exist is dropped, self-edges are dropped, duplicate source→target pairs collapse, and kind is clamped to the six permitted values. Logical relationships (identity, monitoring, dependency) are automatically drawn dashed so you can tell a network path from a conceptual one at a glance.
Layout is applied automatically when the model hasn't produced usable coordinates — if fewer than half the nodes have real positions, the diagram is laid out by tier.

The net effect is a rule that's easy to state and easy to audit: a model can suggest a relationship, but it cannot invent a resource. If a node doesn't correspond to something that genuinely exists in your tenant, it doesn't survive to the canvas.

It's a graph, not a picture

What lands is a typed model, not an image.

Edges carry meaning — six kinds, so the diagram expresses how two things relate rather than merely that they do:

Edge kind	Means	Drawn
depends_on	A needs B to function	dashed
connects_to	A talks to B	solid
data_flow	Data moves from A to B	solid
network	A network-level path (peering, subnet membership)	solid
identity	An authentication / managed-identity relationship	dashed
monitors	A observes B	dashed

Nodes are classified into twelve categories — Web & API, Compute, Containers, Databases, Storage, Integration, Networking, Security & Identity, AI & ML, Monitoring, Analytics, Other — each with its own colour, and into nine architectural layers that run edge → presentation → application → integration → data → networking → security → monitoring → shared. That layering is what lets the canvas arrange itself in tiers like a diagram a human would draw, instead of a hairball.

Groups wrap nodes by subscription, resource group, VNet or tier, which is what makes forty resources legible at all.

Reading the estate back to you

Each node renders as a card that carries rather more than a label. Alongside the name and a category-coloured Azure icon, it can show the hosting model (IaaS / PaaS / SaaS), a rough monthly cost estimate derived from type and SKU, and whether the resource is publicly or privately reachable — three questions an architect asks about every box on a whiteboard, answered on the box itself.

Nodes can also carry badges from two other systems: best-practice lint (single points of failure, missing redundancy) and assessment findings with their severity, so once you've run the Part 4 assessments the diagram becomes a heat map of where your problems physically live. Treat the cost figure as a sizing hint that tells you which corner of the diagram is expensive — not as a bill. Reconciled cost, against Cost Management, belongs to a later post.

The same diagram, six different questions

Because the diagram is a typed model rather than an image, it can be re-read without being redrawn. This is the part that surprises people in a demo, and it's where a live canvas pulls decisively ahead of a Visio file.

Azure view turns on the shared-responsibility lens: every node picks up an IaaS / PaaS / SaaS / Net badge, a globe or padlock for whether it's publicly or privately reachable, its indicative cost, and peering and private-link edges. Anything publicly reachable gains a red ring. It answers "what's exposed here?" in one toggle, on a diagram you didn't have to annotate.

WAF tint recolours the diagram by a failing Well-Architected pillar — Security, Reliability, Cost, Operations, Performance. Switch pillar and the highlight moves. Combined with the live assessment badge in the toolbar, the diagram becomes a map of where your findings physically live rather than a list of resource IDs in a report.

Boundary boxes regroup the whole diagram on demand: no boundaries, or by resource group, subscription, VNet, subnet or region. Watching the same twenty-odd resources rearrange from a flat application view into a subnet view is the fastest way to explain a network design to someone who's never seen it — and it's the clearest possible proof that this is a model, not a picture.

Hosting filter narrows to just IaaS, PaaS, SaaS or network plumbing, which is how you find the three VMs hiding in an otherwise serverless estate.

Impact is blast radius on the diagram: select a node and everything connected upstream and downstream lights up while the rest dims. Path traces the directed request path north-to-south through the system. Between them they answer the two questions every incident starts with — what does this depend on, and what breaks if it goes.

Review collects the best-practice lint into a numbered panel, and the findings also render on the nodes themselves — a Cosmos account with no private endpoint gets flagged in place, next to the resource, not in a spreadsheet.

None of these change the diagram. They're lenses over the same validated model, which is exactly why they're cheap to switch between.

A canvas you can actually work with

The diagram renders on a real graph canvas, so it behaves the way you'd expect rather than like a static export. Pan, zoom, drag nodes into a shape that matches how you think about the system, multi-select, and use the minimap when a diagram outgrows the viewport.

It's a genuine editor, not a read-only render:

Add resources by dragging from a categorised palette of Azure types.
Draw edges by dragging from one node's handle to another.
Edit anything in the inspector — a node's name, type, category, layer and metadata; an edge's label, kind and dashed/solid style; a group's name and kind.
Add groups to carve the diagram into subscriptions, resource groups or tiers.
Add sticky notes for the annotations every real diagram accumulates.

And when your manual dragging has made a mess, six layout algorithms are one click away: top-down hierarchical, left-right hierarchical, tier-layered, grid, radial and force-directed. Watching a diagram reorganise itself between them is genuinely the fastest way to find the view that makes a given system make sense.

Enhance, rebuild, and check for drift

Three buttons keep a diagram alive after the first generation.

Enhance is goal-directed AI refinement: tell it what to improve — "group these by tier", "show the data flows more clearly" — and it revises the existing diagram while preserving what's already correct, running through the same validation pass as a fresh generation.

Rebuild re-reads the live estate and regenerates. Generation runs as a background job, not a frozen browser tab: you get a phase and a percentage — scope → query → ai → save → done — so a big workload taking a couple of minutes looks like progress rather than a hang. Jobs are capped at three concurrent so a bulk rebuild can't stampede your model provider.

Drift compares the diagram against live Azure on demand — the mechanics are in the next section, but it's worth noting it's right there in the toolbar, one click from the picture it's checking.

Two more keep the diagram connected to the rest of the workbench. Ask AI puts a question box against this specific diagram, so "why is this Cosmos account reachable from the internet?" is answered with the architecture already in context. And Investigate hands the whole thing — diagram and its memory — straight into a Part 2 deep investigation, which is the loop this series has been building toward: the map you generated in ninety seconds becomes the context a war room reasons over.

Everything is undoable, and each diagram keeps an Activity log and a History of revisions, so an edit made in a hurry is never a one-way door.

Getting it back out again

A diagram trapped in a tool is a diagram that won't be used, so export is deliberately broad:

PNG and SVG for design docs, change requests and slide decks.
Mermaid to paste straight into a README or a wiki — and it round-trips: you can import an existing Mermaid flowchart back onto the canvas.
JSON for the full model, if you want to do something else with it entirely.
Bicep and Terraform skeletons — scaffolding derived from the diagram. They're a starting point, not a deployable module, and the product says so.

There's also a Present mode that takes the diagram fullscreen, which turns out to be what people actually reach for in an architecture review.

Diagrams have a lifecycle — draft → in review → ready, plus archive — and can be organised into collections, so a tenant with sixty diagrams doesn't become its own navigation problem. Every save snapshots a revision, and deletion is a recoverable Trash rather than an ending.

Honest about what it is

Two caveats the product is upfront about, and so should this post be.

The confidence value is declared by the model, not computed by the system. It's a useful signal about how sure the model was, not an independent measurement of correctness. The thing you should actually trust is the structural guarantee underneath it: every node maps to a real ARM ID, because that part is enforced in code rather than requested in a prompt.

A first pass is a first draft. Inference from properties is very good at the relationships Azure records explicitly — network paths, hosting plans, private endpoints, disks — and weaker at the ones that only exist at runtime, like an app calling an external API by URL. That's exactly why the canvas is editable and why Architecture Memory exists: the generated diagram gets you ninety percent of the way in ninety seconds, and you spend ten minutes adding what only your team knows.

Part three — Memory, drift, and the doc your on-call reads

Architecture Memory — the intent behind the diagram

A diagram tells you what is deployed. What it can't carry is what your team intended — that the second region is deliberately warm-standby rather than half-finished, that the public IP is a known and accepted risk, that when this app misbehaves you check Front Door before you touch the database. Historically that knowledge lives in one senior engineer's head and leaves when they do.

Architecture Memory is a structured document attached to each architecture, spanning 20 sections across six groups:

Group	Sections
Purpose & shape	Overview, architecture pattern, expected flow, components & responsibilities, dependencies
Topology & access	Network topology, identity & access, data & storage
Security & compliance	Security model, compliance & governance
Resilience & performance	Resiliency targets, scaling & performance, critical thresholds & SLIs
Operations	Observability, runbook / operational notes, change management, cost & sizing notes
Risk & diagnostics	Known gaps & risks, known issues & past incidents, diagnostic hints

A brand-new memory is pre-seeded with the seven highest-signal sections rather than twenty empty boxes, and the AI will draft them from the live architecture for you to correct — which is a much easier job than writing from nothing. Every save snapshots a revision (capped at 50 per document, de-duplicated by content signature), so the design history is recoverable.

Where the two pillars snap together

Here's the payoff, and it's the reason Map is worth doing before Assess.

Architecture Memory is injected into the Part 2 War Room as expert context. When a deep investigation has to fit inside a prompt budget, the memory is trimmed by an explicit priority order — expected flow first, then diagnostic hints, known gaps, critical thresholds, security model, resiliency targets, dependencies, components, network topology, observability, known issues. Eleven sections, ranked by operational value, so the most useful knowledge is the last thing to fall out of the prompt rather than the first.

Which means the note you wrote in a calm Tuesday afternoon — "check Front Door health before you look at the database" — is what a specialist agent reads at 2 a.m. while you're asleep. Documenting your architecture stops being a chore you do for auditors and becomes something that measurably improves the next incident.

Drift: when reality wanders off

Once you have a documented design and live inventory, comparing them costs almost nothing — so the workbench does it continuously.

Drift detection matches documented resources against live ones by ARM ID and sorts every resource into one of four states:

State	Meaning
ok	Present in both the design and the live estate
documented_missing	In the architecture, no longer live
live_uncontrolled	Running in your tenant, in nobody's design
inferred	An edge the architecture asserts without a live pair

It produces a drift score — simply the share of resources that reconcile — and if a workload has no architecture at all, the score is null and everything live is flagged uncontrolled, which is itself a finding worth acting on.

live_uncontrolled is usually the column people stare at. That's the resource somebody spun up in a hurry during an incident eight months ago that nobody has thought about since.

The Know-Me doc — and a refusal to make things up

Diagrams serve architects. On-call needs prose: what is this thing, who owns it, what's the SLA, who do I wake up, and what usually breaks.

The workbench generates that Know-Me doc from the architecture and its memory — 13 sections covering the workload overview, the solution architecture, the Azure services and subscriptions in scope, diagnostics and first-look triage, critical thresholds and SLIs, resiliency and DR posture, known issues and proactive callouts, security posture, support and escalation handling, contacts, and data/compliance/cost notes.

The thirteenth section is the one I'd point a sceptic at, because it's a list of everything the AI refused to invent.

The generator is explicitly forbidden from fabricating the facts that matter most: people, SLAs, RTO and RPO values, on-call rotations, customer names, subscription GUIDs, contract and schedule IDs. It doesn't guess and it doesn't produce a plausible-looking placeholder that reads like a fact. It emits a typed marker in the literal form ⟦TODO: <label> | key=<field>⟧, and every one of those markers is collected into that final checklist section for a human to complete. What Azure does know — subscriptions, regions, real scope — is auto-filled and labelled as platform-sourced, so you can see at a glance which facts came from the platform, which were AI-suggested and editable, and which are waiting on you.

Filling them in is deliberately low-friction. Fields render as the right control for their type — segmented buttons for small strict sets like Critical/High/Medium/Low, a typeahead combobox for larger sets like subscriptions or regions, a plain input for free text — and a Guided Fill mode walks you through every incomplete field one at a time, keyboard-driven, so completing a doc is a five-minute pass rather than an afternoon of hunting.

And the refusal is enforced at the door. A doc moves through draft → in review → published, and publishing is blocked until all five required groups have at least one real answer: escalation, resiliency, SLA, contract, and contacts. You cannot publish a runbook whose escalation path is still a placeholder. An on-call document that confidently states a fabricated RTO is worse than no document at all — it's a document someone will believe at 3 a.m. The gate exists so nobody discovers the gap during an incident.

Finished docs export to Markdown or a branded PDF, support inline images, keep revisions, and go to a recoverable Trash rather than vanishing.

Part four — The Estate Graph

Zoom all the way out

Per-workload diagrams answer "how does this application hang together?" The Estate Graph answers the larger question: how does all of it hang together?

It assembles the whole tenant into one interactive graph — 15 node kinds and 14 edge kinds. Nodes run from the tenant connection through management groups, subscriptions and resource groups, down to resources, and across to the artifacts you've built: workloads, architectures, memories and assessment findings. Edges carry the structure (contains, member_of, belongs_to), the modelling relationships (models, documents, has_finding) and the dependency relationships (depends_on, connects_to, data_flow).

It's workload-first by design. You navigate applications, with raw resources underneath, which is the opposite of how most inventory tools present a tenant and much closer to how teams actually think.

Five overlays, one map

Switch on an overlay and the same map answers a different question:

Overlay	Answers
Cost	Where is the money?
Retirement	What is Azure evolving away from, and by when?
Coverage	What's unmonitored, untelemetered or unprotected? (monitoring, telemetry, backup/DR)
RBAC	Who holds privileged access where?
Change	What moved recently?

That's the argument for a single graph over five separate dashboards: the cost hotspot and the coverage gap and the privileged principal are all the same node, and seeing them on one map is how you notice they're related.

Because a tenant-wide graph can get genuinely enormous, everything is capped on purpose rather than allowed to melt the browser: findings collapse into a single "⚠ N findings" super-node past 18 per workload, expanding a group loads at most 400 resources, and the overlays are bounded too (12 privileged principals, 20 retirement items, 40 change events). You choose a layout — organic, hierarchy, breadthfirst or concentric — and it persists for your tenant.

Every node is a doorway

Select a node and the inspector opens a dossier: for a workload, its environment, criticality, resource count, linked architecture, risk score and findings; for a resource, its type, SKU, location, owner, cost and the workloads it belongs to; for an architecture, its state, AI confidence and how much of its memory is filled.

From there you can trace rather than just look — expand a node's neighbours, highlight a blast radius, or ask for the path between two nodes. And the "open in" actions carry the workload context with you into Inventory, the architecture, its memory, an assessment run, Change Explorer, RBAC, Telemetry, Backup & DR — or straight into a Part 2 war-room investigation. You land somewhere useful, already scoped, instead of at the top of another dashboard.

Read-only, and why that's a design constraint rather than a limitation

Everything in this post is a read path. Workloads, architectures, memories, Know-Me docs and the Estate Graph all write to the Azure Support Agent's own storage — never to Azure. The reads themselves go straight to Azure Resource Graph and the ARM APIs.

That's a deliberate boundary. Mapping is the activity you want to be able to run against production on a Friday afternoon without a change request, without a second pair of eyes, and without thinking twice. The moment mapping can mutate, it acquires a blast radius and people stop running it — which defeats the point. Writes exist elsewhere in the workbench (applying owner tags, for instance), and they're approval-gated and audited when they do. They're a later post.

And if you haven't connected a tenant yet, demo mode seeds the whole pillar with realistic synthetic workloads, architectures and inventory, so you can walk the entire flow before pointing it at anything real.

How it's judged

Health scores are composite and explainable. Seven weighted signals, published bands, and an API that returns which signals contributed, which are missing, and the exact weights used.
Unmeasured never masquerades as bad. Signals that haven't been analyzed are excluded from the denominator rather than scored as zero, and the weights are yours to tune.
Autopilot is review-before-apply. Proposals carry reasoning and confidence, and are never persisted without an explicit human save.
Every node maps to a real resource. Model-emitted ARM IDs are validated against live inventory; duplicates collapse and orphaned edges are dropped before anything renders.
Large estates degrade gracefully. Discovery caps, property budgets and graph caps are explicit and disclosed rather than silently sampling.
The generator declines to invent. People, SLAs, RTO/RPO, contacts, GUIDs and contract IDs become explicit ⟦TODO⟧ markers, and publishing is blocked until all five required groups are genuinely complete.
Drift is measured, not asserted. Documented and live inventory reconcile by ARM ID into four named states with a score you can track over time.
Overlays reflect real data and deep links resolve. Cost, retirement, coverage, RBAC and change all read live sources, and every node opens the artifact behind it.
Read-only throughout. Mapping your estate never changes it, and the Azure Support Agent runs in your own Azure Container App against Azure's own APIs.

Try it yourself

If you've already deployed the Azure Support Agent, this is the fastest way to feel the difference — about ten minutes:

Run Autopilot against one subscription and accept a single proposed workload.
Hit Generate architecture and watch it assemble. Seeing a correct diagram of something you own build itself out of Azure Resource Graph is the moment this stops being abstract.
Open its Architecture Memory and write three honest sentences about how it's supposed to work — the expected flow, one known gap, one diagnostic hint.
Now run a Part 2 deep investigation against that workload.

The difference between the answer you get in step 4 and the answer you'd have got before step 3 is the entire argument for this pillar. The map isn't documentation you produce for someone else's benefit; it's context that makes every other answer better.

No tenant connected yet? Demo mode runs the same flow on synthetic data, with nothing to risk.

Next time — Part 4: Assess — coverage and your real posture. With the map in place, we go looking for trouble on purpose: Azure Monitor baseline alert coverage and whether the alerts you already have would actually fire, telemetry gaps, backup and DR posture with real costs attached, Well-Architected reviews at scale, FMEA risk registers, and who can do what across your estate — each judged against a versioned baseline, and each arriving with the infrastructure-as-code to close the gap.

It's open source and free to run. Three ways to start, in rising order of commitment:

Explore demo mode — synthetic data, no connection required, nothing to risk.
Deploy to your tenant — one click to Azure Container Apps; private networking; your data never leaves.

Or main repository https://github.com/zmustafa/AzureSupportAgent

Building Resilient Power Automate Flows That Stay Running

mlotorto — Thu, 30 Jul 2026 19:04:54 GMT

If you've spent any time working with Power Automate in enterprise environments, you've seen it happen. A business-critical flow fails on a Friday afternoon. The person who built it left the company six months ago. Nobody gets an alert. Nobody knows until Monday morning when someone notices the reports didn't run. By then, the blast radius is already bigger than it needed to be.

Flow failures aren't inevitable, but unplanned ones usually share the same root causes: personal account dependencies, missing error handling, hardcoded configurations, and no visibility into what's actually happening at runtime. The good news is that most of these are solvable with patterns that aren't especially complex, they just require intentionality.

This post covers 16 recommendations across 5 focus areas that I've found to be the highest-leverage investments for making Power Automate flows more resilient, performant, and easier to operate at scale. I've ordered them roughly by the impact you'll see fastest, with supporting documentation for each section for further reading and implementation guidance.

1. Credential and Connection Resilience

This is where I always start with customers, because it's the most common source of outages and also one of the easiest to fix systematically.

Migrate high-value flows to service principal ownership. Flows owned by user accounts fail when passwords change, MFA prompts fire, or the user leaves the organization. A service principal is a non-human identity that represents an application or service and doesn't have any of those human lifecycle problems. The pattern is to register an Azure AD app registration per workload domain, grant it the required API permissions, and bind the flow to it using connection references rather than a personal connection. For connectors that don't yet support service principals (SharePoint, Outlook, Teams), use a hardened service account protected with Privileged Identity Management.

Service principal owned flows

Service Principal Licensing Requirements

Resolve broken connections

Use connection references for all connections. Embedded personal connections break when flows are exported, imported, or the owner changes. If you're building flows inside solutions, there's no good reason not to be using connection references. Create one per connector type used in your solution and bind every flow to the reference rather than a personal connection. When you deploy through environments, you update the binding once rather than hunting through individual flows.

Use connection references in solutions

Pre-populate connection references for CI/CD

Store secrets in Azure Key Vault, not flow variables. Hardcoded API keys or tokens in flow expressions show up in run history and are a real security exposure. The Key Vault connector has a "Get secret" action you can drop at the top of any flow. Restrict access to the service principal only, and when you rotate a secret in Key Vault, the flow picks it up dynamically with no changes needed.

Azure Key Vault connector reference

Key Vault secrets with environment variables

Secure data in cloud flows

2. Error Handling and Retry Logic

Unhandled errors in Power Automate produce silent failures. The flow shows red, maybe an email goes out if you're lucky, and whoever is on call has to play detective with limited context. Building proper failure boundaries before flows reach production is one of the highest-ROI things you can do.

Wrap every action in a Scope with "Configure run after." A Scope action groups a logical block of actions and lets you branch on "Has failed." The pattern is straightforward: wrap each logical unit (get data, send notification, write to Dataverse), add a parallel branch configured to run on failure, and in that branch log the error, send an alert, and terminate cleanly. This keeps failure handling isolated and predictable rather than having it bleed across the entire flow.

Use scopes to organize actions

Best practices for using scopes

Employ robust error handling

Enable built-in retry policies on connector actions. Transient HTTP and connector errors are a fact of life. The retry policy setting on any connector action handles them automatically without custom logic. Exponential interval is generally preferable over fixed. Four retries with a PT10S interval is a reasonable starting point. One important caveat: check for idempotency before enabling retries on Dataverse writes, since a write that partially succeeded before a transient error can create duplicate records if blindly retried.

Implement a retry policy

Handle transient faults (Well-Architected)

Flow limits & config (retry settings)

Use "Do until" loops with exit conditions for polling. A polling loop without bounds can run indefinitely or silently miss the exit condition it was waiting for. Set a maximum iteration count and a timeout, and make sure your exit condition is the expected success state and not just the absence of error.

Until logic documentation

Flow limits: concurrency & looping

Capture and surface error details in failure notifications. Generic failure emails like "Your flow failed" are nearly useless for triage. In the failure branch, use workflow().run.name and actions('ActionName')?['error'] expressions to pull the actual error message, and construct a direct link to the run using the environment name, flow ID, and run ID. Sending this as a Teams adaptive card with structured fields makes triage significantly faster than sifting through the admin center.

Understand error codes

3. ALM and Environment Strategy

Environment drift is sneaky. It builds up over time through ad-hoc edits made directly in production, hardcoded values that break on promotion, and manual export/import processes that are just inconsistent enough to cause problems at the worst times.

Deploy only managed solutions to production. Unmanaged solutions in production allow anyone with access to make direct edits that bypass your ALM process. The model I recommend is straightforward: build in Dev with unmanaged solutions, export as managed for promotion, validate in Sandbox, and promote to Production through Pipelines. Use the "Block unmanaged customizations" setting in Environment security policies to enforce this at the platform level.

Solution concepts with Power Platform

Create and update custom solutions for ALM in Power Platform

Use Power Platform Pipelines for automated promotion. Manual export and import is error-prone and doesn't scale. Pipelines automate the full Dev to Sandbox to Production path, support pre-deployment validation before each promotion, and let you restrict production deployment approvals to designated owners. If you're not using Pipelines yet, this is worth the setup investment.

Overview of pipelines in Power Platform

Set up pipelines in Power Platform

Deploy pipelines as a service principal

Eliminate hardcoding with Environment Variables. Flows that reference hardcoded SharePoint site URLs, list names, or IDs break every time they're promoted to a new environment. Replace them with environment variables, set the values at the environment level rather than in the solution itself, and validate bindings as part of your pre-deployment checklist.

Environment variables for Power Platform overview

Use environment variables in Power Automate solution cloud flows

Apply DLP policies scoped per environment. Overly permissive DLP in production is a real risk for data leakage and flow instability. Classify connectors intentionally across Business, Non-Business, and Blocked. Scope policies from the top down, tenant-level first and then environment-level. Use endpoint filtering to restrict rather than wholesale block where possible, and always test DLP policy changes in Sandbox before promoting them to production.

Data loss prevention policies overview

Data policies for Managed Environments

Identify blocked Power Automate flows from data policies

4. Monitoring and Observability

If you don't have visibility into flow health, you're finding out about failures the worst possible way: from users. Most organizations have significantly less monitoring in place for Power Automate than they do for their traditional application workloads.

Enable Power Platform admin center flow analytics. This is the quickest win in the monitoring category. In the Power Platform admin center, navigate to Analytics and enable flow run history and error reporting at the environment level. From there, set up a regular cadence (weekly is a good start) to review failure rates for flows classified as business-critical. Having that baseline gives you something to measure against when you start making resilience improvements.

Tenant-level analytics

Best practices for monitor your flows (analytics)

Integrate with Application Insights for deeper observability. For environments running Managed Environments, you can configure the Data Export setting in the admin center to push cloud flow runs, triggers, and actions to an Azure Application Insights resource. Once data is flowing, you get full query and visualization capability for run health, custom diagnostics, and alerting. This is the right long-term destination for any organization that takes operational reliability seriously.

Overview of integration with Application Insights

Set up Application Insights with Power Automate

5. Performance and Throttling

High-volume flows that aren't tuned for throttling will eventually hit API rate limits and start failing or queuing in ways that are hard to predict. Most of the common patterns here are straightforward to address.

Apply concurrency control on trigger-heavy flows. By default, Power Automate concurrency is unbounded. A high-volume trigger can spin up thousands of parallel runs simultaneously and hit API limits quickly. Enable concurrency control on the trigger, set the degree of parallelism based on your connector's rate limits (starting between 10 and 25 is reasonable), and monitor queue depth in the Power Automate portal to tune from there.

Understand limits & avoid throttling

Flow limits: concurrency, looping, debatching

Batch SharePoint and Dataverse operations instead of looping. An "Apply to each" loop that makes an individual API call per item scales linearly in both API call count and runtime. For SharePoint, use the "Send an HTTP request to SharePoint" action with OData $batch to group reads and writes. For Dataverse, use batch actions or FetchXML to pull multiple records in a single call. As a rough target, if a loop is making more than 50 individual API calls per run, it's a good candidate for batching.

Make batch requests with the REST APIs - SharePoint

Execute batch operations by using the Web API - Dataverse

Use chunking for large file or record sets. Flows processing large payloads can hit the 100MB message size limit or simply time out. The pattern here is to split large workloads into pages using OData $top and $skip or Dataverse paging cookies, use a parent flow to orchestrate chunked child flows, and log progress state to Dataverse so the parent can resume if a chunk fails.

Handle large messages in workflows using chunking in Azure Logic Apps (applies to Power Automate)

Message size limits in Power Automate

Where to Start

If you're looking at this list and wondering where to begin, the recommended implementation order is:

Credential and Connection Resilience first. This eliminates the most common source of outages. Migrate to service principals where supported and harden service accounts with PIM everywhere else.
Error Handling and Retry Logic second. Low effort relative to the impact. Adding scopes, run-after branches, and retry policies is mostly configuration work, and the payoff is immediate.
ALM and Environment Strategy third. Managed solutions in production, Pipelines for promotion, and environment variables in place of hardcoded values.
Monitoring and Observability fourth. Structured logging and failure alerting gives you quantifiable before-and-after visibility on everything else you're doing.
Performance and Throttling last. Address this for high-volume flows once the foundational reliability work is in place.

You don't need to implement all 16 of these at once. Even getting the first two focus areas locked down will materially reduce the failure rate for most enterprise Power Automate implementations. The goal is to build flows that run reliably, fail visibly when something goes wrong, and don't require a forensic investigation every time they do.

Start there and layer in the rest over time.

Author: Marc Lotorto Contributors: Sheldon Dsouza, Ben Kohler

Microsoft Foundry Models: A Field Guide for Cloud Solution Architects

luciacasucci — Wed, 29 Jul 2026 13:06:57 GMT

Which model families matter, what each one is for, and how to choose through the Well-Architected lens.

July 2026. Views are my own, not Microsoft guidance. Model availability, status tags, and capabilities change fast: everything below is accurate at time of writing. Verify against the live catalog before you commit an architecture.

Foundry in one minute

Microsoft Foundry is the model platform on Azure: one catalog, one control plane. What that means in practice:

1,900+ models across OpenAI, Anthropic, Mistral AI, Meta, DeepSeek, xAI, Cohere, NVIDIA, and others (at time of writing)
Two deployment modes: serverless APIs (pay per token, fastest path) and managed compute (your infrastructure, your controls)
One governance plane: Entra ID for identity, Azure billing with existing commitments applying, Azure Monitor for observability, Azure Policy for guardrails

The implication for architects: adding a model vendor is no longer a procurement cycle. It's a routing decision. That changes how you should design, and it raises the bar on the plumbing: gateway, model service layer, config-driven model names.

The lens: five pillars, five questions

The Azure Well-Architected Framework gives you five pillars. Applied to model selection, they become five questions:

Cost Optimization: does this workload need frontier pricing, or is a smaller model good enough?
Reliability: what happens when this model retires, throttles, or degrades?
Performance Efficiency: what latency does the user experience at p99, not on average?
Security: who can call this model, and what filters sit in front of it?
Operational Excellence: can you swap this model without an engineering sprint?

Each vendor section below closes with the pillar notes that matter most for that family.

[Diagram 1 – The Model Catalogue Toolbox]

OpenAI: tiers, not versions

The GPT-5 family in Foundry is not one model with version numbers. It's tiers with different jobs (at time of writing):

Deep reasoning (gpt-5.6-sol): multi-step analysis, complex agentic workflows, hard coding problems. Slowest and most expensive. Use it where being wrong is expensive.
Frontier workhorse (gpt-5.5): document work, coding, data analysis, 1M-token context. Where most complex production traffic belongs.
Chat tier (gpt-5.5 Instant): multi-turn assistants, tool orchestration, retrieval-grounded apps. Faster, cheaper, precise enough.
Nano tier (gpt-5 nano): ultra-low-latency Q&A at volume. When a user is waiting and the task is simple, this is the answer.

WAF notes

Cost + Performance: The most common failure mode is routing everything to the biggest model. Users wait longer, the bill grows, and quality doesn't improve because most requests never needed the depth.
Reliability: Every deployed model has a retirement date. Check it at design time and plan the migration path before you need it.
Operational Excellence: Abstract model names behind config. A retirement should be a config change, not a sprint.

Anthropic: Claude, now GA on Azure

Claude Opus 4.8, Claude Sonnet 5, and Claude Haiku 4.5 are generally available in Foundry hosted on Azure (GA announced June 29, 2026; Sonnet 5 followed in early July). Same Azure billing, same Entra ID, no separate Anthropic contract. Claude Fable 5, Anthropic's frontier agent model, returned to availability on July 1 after an export-control suspension from June 12-30. Mythos-class models remain gated research preview for approved organizations only (at time of writing).

Opus 4.8: maximum reasoning depth. Long-horizon agentic runs, hard multi-step analysis, complex coding.
Sonnet 5: the default for agents and coding work. Strong tool use, large context, handles workloads that used to need Opus-class models.
Haiku 4.5: the volume tier. Classification, tagging, metadata, simple Q&A at thousands of calls.

WAF notes

Security: Unlike Azure OpenAI models, Claude deployments ship with no built-in content filtering at deployment time (at time of writing). Configure Azure AI Content Safety or gateway-level filtering yourself. This is the single most-missed item in reviews.
Reliability: A safety-flagged request returns stop_reason "refusal" with HTTP 200, not an error status. Handle it in application logic, not in status-code checks.
Operational Excellence: Foundry does not pass through Anthropic's native rate-limit response headers. Use Azure Monitor for consumption tracking. And verify context-window figures in the portal's deployment pane: public documentation sources have disagreed.

Mistral AI: open weights and specialists

Mistral's lineup answers requirements the flagships don't (at time of writing):

Mistral Large 3: frontier-class capability with open weights. Tool calling, long context, agentic workflows. Deployable on managed compute when serverless doesn't fit the compliance story.
Mistral Document AI: OCR and extraction for scanned PDFs, forms, and technical manuals, feeding straight into RAG pipelines.
Codestral: purpose-built code generation.
Ministral: small models for edge and low-latency scenarios.

WAF notes

Security: Open weights plus deployment control is the answer when data residency or model inspection is a requirement, not a preference.
Cost: On document pipelines, a dedicated extraction model routinely beats prompting a general model, on both accuracy and price.

Meta Llama: the open ecosystem

Llama spans 1B and 3B small models for edge and on-device inference up to 405B for synthetic data generation and distillation. It has the widest open fine-tuning community, which matters when you need domain adaptation with full control of weights and training data.

WAF notes

Cost + Performance: Llama SLMs handle classification, extraction, and summarization at a fraction of frontier pricing, and at the edge they remove network latency entirely.
Operational Excellence: Managed-compute deployments put patching, scaling, and monitoring on your team. Budget for the ops, not just the tokens.

DeepSeek: reasoning on a budget

DeepSeek's family includes reasoning models trained for step-by-step math, science, and coding tasks at aggressive price points. Worth evaluating wherever reasoning depth matters but flagship pricing doesn't fit.

WAF notes

Cost: Benchmark against your actual reasoning workloads. The price gap to frontier reasoning tiers is large enough to justify the evaluation effort.
Security: Same rule as all partner models: bring your own content filtering, and check data-handling terms on the model card before production use.

xAI Grok and Microsoft Phi

Grok: reasoning and non-reasoning models aimed at data extraction, coding, summarization, and agentic applications.
Phi: Microsoft's small-model family, 60M+ downloads (at time of writing). On-device, edge, and high-volume simple tasks.

WAF notes

Performance: Phi at the edge removes the network round trip entirely. For latency-critical simple tasks, no hosted model competes with no network call.

The selection checklist

Before any model earns a production slot, walk it through the five pillars:

Pillar	What to check before you commit
Cost Optimization	Map the workload to a tier before picking a name. Price the small-model alternative for every high-volume task. Check reservation options for steady traffic.
Reliability	Find the retirement date before the first deployment. Plan the migration path (blue/green beats in-place for most production workloads). Know the throttling behavior and your overflow lane.
Performance Efficiency	Benchmark p95/p99 latency with your prompts, not the vendor's. Enable streaming for anything user-facing. Consider edge or SLM options where network latency dominates.
Security	Verify what content filtering ships by default; for most partner models the answer is none, so configure it. Use Managed Identities, not API keys. Check data-handling terms per model card.
Operational Excellence	Keep model names in config, never in code. Put a gateway and a model service layer in front. Define promotion gates (evals, latency budgets) before the next version ships.

The pattern that ties it together

Your most expensive model should handle your fewest requests. The hybrid pattern:

Frontier model as the brain: complex decisions, planning, orchestration. Low volume, high stakes.
Small models as the hands: Phi or Llama SLMs for classification, extraction, and summarization at volume.
Specialists where they win: reasoning models for math-heavy tasks, extraction models for documents.
A router in front: the gateway decides which request goes where. The application never knows.

The catalog isn't a leaderboard. It's a toolbox. The teams that get cost, latency, and reliability right pick per workload, not per benchmark, and they build the plumbing (gateway, model service layer, config-driven names) that makes multi-model architectures operational instead of aspirational.

[Diagram 2 – The pattern that ties the models together]

Cheat sheet: the catalog briefly

One row per family. Use it as the 30-second version of everything above (all entries at time of writing).

Family	Key models	Reach for it when	Watch out for
OpenAI GPT-5	gpt-5.6-sol, gpt-5.5, gpt-5.5 Instant, gpt-5 nano	Default frontier lineup; tiered from deep reasoning to low-latency nano	Routing everything to the flagship; retirement dates; hard-coded names
Anthropic Claude	Opus 4.8, Sonnet 5, Haiku 4.5 (GA on Azure); Fable 5	Agentic and coding workloads; second frontier vendor for resilience	No default content filtering; refusals return HTTP 200; no native rate-limit headers
Mistral AI	Large 3, Document AI, Codestral, Ministral	Open weights, data residency, document extraction pipelines	Managed compute shifts ops burden to your team
Meta Llama	1B-3B SLMs up to 405B	Fine-tuning with full weight control; edge inference; distillation	You own patching, scaling, monitoring on managed compute
DeepSeek	Reasoning model family	Step-by-step math, science, coding where flagship pricing doesn't fit	Bring your own filtering; check data-handling terms per card
xAI Grok	Reasoning + non-reasoning models	Data extraction, summarization, agentic apps	Same partner-model filtering rule
Microsoft Phi	Small-model family	On-device and edge; high-volume simple tasks	Capability ceiling; validate quality before routing volume

Disclaimer

The views expressed in this article are those of the author and do not necessarily reflect the official policy or position of Microsoft. The author is a Microsoft employee.

References

Microsoft documentation and announcements:

Vendor announcements:

Building an AI Azure Ops Workbench — Part 2: The War Room — watch AI root-cause a 5xx in 90 seconds

zmustafa — Thu, 30 Jul 2026 21:20:40 GMT

In Part 1 we made the case for the Azure Support Agent, an operations copilot that runs in your own tenant and reasons over live evidence through Azure's own APIs. This post opens the first pillar — Converse — and goes inside the two features that make it real: a chat that actually queries your estate and shows its work, and the marquee act, a War Room of specialist agents that fan out in parallel to turn "why is the site throwing 5xx?" into a ranked, validated root cause. Let's watch it work.

Every incident starts the same way: a question you can't answer fast enough. The signal you need is almost always already in Azure — the metrics are in Azure Monitor, the topology is in Azure Resource Graph, the recent changes are in the activity log, the expiring secret is in Key Vault. The hard part at 2 a.m. is that the answer is spread across a dozen surfaces and one pair of hands. The Converse pillar is about collapsing that distance: you ask in plain English, and the agent does the querying, correlating and explaining for you — against your live estate, not a training-data guess about how Azure works.

First, a chat that reads your tenant

Before we get to the marquee demo, it's worth being precise about what "chat" means here, because it is not a toy wrapped around a language model. When you ask "Why is the checkout API throwing 5xx?", the Azure Support Agent answers by doing the work — it reaches into your subscription through the official Azure MCP server (and, when you've connected it, a Microsoft Graph / Entra ID MCP server), runs read-only queries against Azure Resource Graph, Monitor and the rest, and streams the whole thing back to you as it goes.

That streaming detail matters more than it sounds. Answers arrive over Server-Sent Events, so you don't stare at a spinner — you watch a live timeline assemble: the model's reasoning as it narrates its plan, then each tool call it makes (the query it ran, how long it took, and a one-line summary of what came back), step by step, until the final answer lands. When it's done, that timeline doesn't vanish. It's persisted with the message, so if you reload the page a week later the reasoning, every tool call, and any screenshots you pasted in are all still there. Nothing about how the agent reached its conclusion is thrown away — which is exactly what you want when you have to defend that conclusion to a change board later.

A few things make it comfortable to live in day-to-day:

Multiple sessions, isolated context. Keep three investigations open at once; each chat carries its own model choice, its own Azure connection, and its own optional workload scope.
Vision input. Paste a screenshot of a failing blade or an error toast; it's part of the question and it's saved with the turn.
Starter suggestions. Not sure how to phrase it? Curated prompts, grouped by domain — connectivity, identity, security, compute, performance, cost — get you moving with one click.
Cancel-safe turns. This is the quietly brilliant part. Hit Stop, or just close the tab, and the work keeps running server-side. The turn is decoupled from your browser connection entirely: it finishes, saves its partial or full answer and its timeline, and is waiting for you when you come back. Reconnect and the stream replays what you missed and picks up live. Your investigation is never hostage to a flaky Wi-Fi connection or an accidental refresh. In Deep mode it goes one further: the investigation tree is checkpointed to the database as it builds, so even a mid-flight reconnect picks up every hypothesis already resolved.

That last point is the difference between a demo and a tool you'd actually trust mid-incident. The work is durable because the evidence is durable.

Then, when it's genuinely hard: the Azure Support Agent War Room

A single model narrating one line of investigation is perfect for "which NSG is blocking this port?" But real incidents are rarely one line of investigation — they're five plausible theories that each need chasing before you can rule them out. That's more parallel work than one reasoning thread can do well, and it's exactly where a single-threaded chatbot starts guessing.

So you flip on Deep mode, and a War Room convenes.

Instead of one generalist, the agent fields a bench of eight specialists, each with its own lens on the estate:

Specialist	What it owns
🌐 Networking	NSGs, routes, DNS, private endpoints, peering, connectivity
🔑 Identity & Access	RBAC, Entra ID, managed identities, Key Vault, secrets
⚡ Compute & Apps	VMs, App Service, AKS, Container Apps, Functions, scale sets
💾 Storage & Data	Storage accounts, SQL, Cosmos DB, disks, backups
🔐 Security & Exposure	Public exposure, Defender, open ports, permissive NSG rules
📈 Reliability & Performance	Resource health, availability, metrics, scaling, SLA
💰 Cost & Governance	Spend anomalies, Azure Policy, quotas, tags, budgets
📊 Monitoring & Logs	Activity logs, alerts, Application Insights, Log Analytics

The point of the bench isn't role-play — it's parallelism with focus. Several specialists research at the same time, each pulling live evidence for its own domain, so a broad question gets broad coverage in the time a single thread would spend on one branch. Here's what that looks like mid-flight — eight specialists working the estate at the same moment, each on its own lane:

You pick the bench — or let it pick for you

You're not stuck with all eight specialists on every question. Before a Deep investigation runs you can hand-pick the bench — just Networking and Identity for a connectivity puzzle, say — or let the agent suggest the roster for you. The suggestion isn't a black box: it proposes a recommended set with a one-line reason for each pick ("Identity — the error is a 403 and the app uses a managed identity"), and if the model is ever unsure it falls back to deterministic keyword matching, so you always get a sensible default. Pick none and it staffs the room itself.

Why bother narrowing it? Focus. A DNS question doesn't need the Cost specialist spending tool calls on spend anomalies. A tighter bench means less noise in the tree, a faster answer and lower token cost — without giving up the parallelism that makes the War Room worth it in the first place.

How an investigation actually unfolds

Under the hood the War Room runs a disciplined, four-phase play — and because it streams the whole way, you watch each phase happen rather than waiting for a black-box verdict:

Incident research. The agent spends a bounded burst of read-only tool calls building context on the live estate, then proposes a short slate of two to three distinct hypotheses, each attributed to the specialist best suited to test it. No fishing expeditions — the work is time- and iteration-boxed so it converges instead of wandering.
Hypotheses formation. The candidate root causes are pinned down as a clean set — parallel theories, not a single lucky guess — ready to be tested independently.
Validating hypotheses. Now the specialists go to work, in parallel, each running its own read-only investigation to either confirm or kill its assigned theory. Every hypothesis gets a verdict — validated, invalidated, or inconclusive — backed by a paragraph of specific evidence. A theory that survives can spawn one deeper sub-hypothesis, so the investigation drills down where the evidence leads and drops the dead ends. The result is a compact hypothesis tree (a couple of levels deep, deliberately capped so it stays readable) that records not just what was true but what was checked and ruled out.
Conclusion. With the tree resolved, the agent streams a plain-English answer — the root cause, the evidence chain that proves it, a severity, and concrete next actions — and attaches a confidence score. That score is computed, not asserted: three quarters of it comes from how the strongest hypothesis resolved and one quarter from how the whole slate resolved, so one clean validation outranks three shrugs, and an invalidated theory contributes nothing. A single validated root cause reads very differently from three inconclusive maybes, and the number says so.

Two design choices keep that four-phase play fast and honest rather than open-ended:

It's boxed, not boundless. Every phase carries both an iteration limit and a wall-clock budget, each agent's tool calls fan out only so wide, and the hypothesis tree is hard-capped in depth, breadth and total nodes. That's deliberate: an investigation that can run forever is one that runs up a bill and wanders. The budgets force it to spend its evidence-gathering where it counts and then converge — you get an answer in about a minute and a half, not a research project.

Those aren't vibes; they're constants. The guardrails, concretely:

Guardrail	Value
Hypotheses formed per investigation	2–3
Tree depth	2 — a hypothesis, plus at most one sub-hypothesis under it
Validated nodes across the whole tree	5
Research phase	≤ 5 tool-loop iterations, ~55 s wall clock
Each hypothesis validation	≤ 3 iterations, ~40 s wall clock
Concurrent tool calls inside one turn	6
Hypotheses validated in parallel	up to 12 (configurable; parallelism can be turned off entirely)

Read that table as a cost control as much as a latency one. A wide parallel search across a live tenant is exactly the kind of thing that quietly burns tokens and API quota, so the ceilings are part of the product, not a tuning afterthought.

Fast where it's cheap, strong where it counts. The broad research sweep and the parallel hypothesis validation can run on a fast, inexpensive model, while the final conclusion — the part you actually read and act on — is always written by your strongest configured model. You get the breadth of a wide parallel search without paying premium-model rates for every intermediate tool call. The fast tier is opt-in: name a cheaper provider and model for the intermediate phases and they're used automatically; leave it unset and the whole investigation runs on your primary model.

Ninety seconds after you asked, you have a conclusion rather than a hunch. Here is that exact claim, unedited, against a live tenant: one in five requests is failing with a 5xx and nothing was deployed. The room convenes, eight specialists fan out, and the verdict comes back that nothing is broken at all — the 5xx responses are being minted by the Container Apps ingress in front of an app pinned to a single replica, while the loud suspects (crash loop, autoscale churn, database exhaustion, storage) are each cleared with a number attached. And the whole thing — every hypothesis, its verdict, its evidence, the specialist who tested it — is saved as a navigable hypothesis tree attached to the chat. The next person on call doesn't start from zero; they start from your investigation.

The rest of this post is about how that happened — and there are three more recorded runs further down, including one with no incident at all.

Specialists don't just read metrics — they run real diagnostics

It's tempting to picture the bench as eight things running Resource Graph queries. They do that, but the more interesting part is that a specialist can reach for higher-order diagnostic tools mid-investigation — the same ones a senior engineer would open — all still strictly read-only:

Performance Profiler. A specialist can launch an Azure Monitor-threshold performance profile on a workload and fold the resulting scorecard and ranked bottlenecks straight into the evidence, rather than eyeballing raw metric charts.
Access and ownership lookups. "Who can actually reach this resource?" and "who owns it?" are answered from the workbench's cached RBAC and ownership scans — real least-privilege evidence, without hammering live Azure mid-incident.
In-guest sandbox checks. Where a workload has onboarded sandbox VMs and you have the permission, a specialist can run read-only in-guest diagnostics — resolve a DNS name from inside the subnet, test a connection — the checks you'd normally SSH in to run by hand.

The point is that the War Room investigates with the same toolbox you would, only in parallel and without the tab-hopping. And because Deep mode is read-only end to end, none of those tools can change a thing — a profiler runs, a VM command reads, but nothing mutates your estate.

Two investigations from the field

The single-replica case above is a clean one-liner once you see it. The real value of the War Room shows up when the obvious suspect is innocent and the truth is hiding one layer over. The next two cases are composites — teaching examples assembled from the shape of incidents I've worked, not transcripts of a single run — chosen because they show that pattern better than any one real investigation does. (If you want unedited runs, they're in the section right after this one.) In both, notice the pattern: every signal was already in Azure. The win isn't new data; it's several specialists correlating across domains at once, and a tree that records what got ruled out, not just what was true.

Investigation 1 — "The database is slow" (that isn't the database)

A checkout API starts returning intermittent 500s and stalling under peak load. On-call's gut says the SQL database is maxed out — so that's the first thing the War Room checks, and it's the first thing it rules out. The symptom points at the data layer; the cause is one layer over, and it only shows itself under load, which is exactly why a quiet repro never reproduces it.

#	Specialist	Hypothesis	Verdict	Evidence
H1	💾 Storage & Data	Azure SQL DTU/vCore saturation or blocking	Invalidated	DB sitting at ~30% DTU, no blocking waits, query stats healthy
H2	📈 Reliability & Perf	App Service plan CPU/memory saturation	Inconclusive	CPU normal — but active connection count is abnormally high
H3	🌐 Networking	SNAT port exhaustion on outbound connections from App Service	Validated	`SnatConnectionCount` hits its ceiling exactly when the 500s spike; failed-connection metric climbs under load
H3.1	🌐 Networking (sub)	App opens a fresh connection per request instead of pooling	Validated	Outbound connection churn tracks the request rate almost 1:1

The conclusion streams back in plain English: the app is exhausting its SNAT ports because it opens a new outbound connection per request instead of pooling them — so under load it simply runs out of source ports and requests start failing. The database was never the problem. Fixes follow, ranked: enable connection pooling in the app, add a NAT Gateway for outbound scale, or move SQL behind a private endpoint so those calls don't consume SNAT at all. Confidence: high — one cleanly validated chain, and the loud suspect explicitly cleared.

Here's the part worth sitting with: every number in that table lived in Azure Monitor the whole time. What was missing at 2 a.m. wasn't data — it was someone with the bandwidth to check the data, the compute and the networking layers simultaneously and notice that the SNAT ceiling, not the DB, lined up with the failures. That correlation is the whole job, and it's what the War Room does in parallel.

Investigation 2 — "Random 403s after a harmless change"

Overnight, roughly 5% of requests to an internal API start returning 403 Forbidden. It's intermittent, and it only hits some instances — the classic "works on my pod" ghost that resists every local repro. A 403 screams identity, so that's where a single-thread chatbot would dig in and stay. The War Room checks identity too — and clears it in the first pass.

#	Specialist	Hypothesis	Verdict	Evidence
H1	🔑 Identity & Access	App registration secret/cert expired	Invalidated	Credential valid for 60+ more days
H2	🔐 Security & Exposure	A new NSG / WAF rule is blocking traffic	Invalidated	No matching deny; rule set unchanged in the window
H3	📊 Monitoring & Logs	Correlate the 403 onset to a recent change	Validated (lead)	Change history shows a Private Endpoint + Private DNS zone added yesterday
H4	🌐 Networking	One spoke VNet is unlinked to the new Private DNS zone, so its instances resolve Key Vault/Storage to the public IP and hit the resource-firewall deny	Validated	The failing requests map exactly to instances in the unlinked subnet; resource-firewall logs show denied public-IP calls

The root cause: a private-endpoint rollout the day before left one spoke VNet unlinked to the new Private DNS zone. Instances in that subnet keep resolving Key Vault and Storage to their public endpoints, and the resource firewall — correctly — denies them. Everyone else is fine, which is why it looked random. The fix is a single missing VNet link. Confidence: high — a change correlated to the onset, and subnet-precise evidence tying the 403s to the gap.

This is the investigation that takes a human an hour of cross-referencing an activity log against a network diagram. In the War Room the Monitoring specialist surfaced the suspicious change while the Networking specialist proved the DNS gap — at the same time — and the tree shows the two innocent identity and security theories that got cleared along the way, so nobody re-litigates them later.

It's also a preview of where this series goes. That "intended topology: all spokes linked" is exactly the kind of fact the workbench remembers as Architecture Memory (Part 3), the missing link is the sort of thing drift detection flags before it pages anyone, and the "what changed last night" answer is Change Explorer doing in one query what used to be an hour of log archaeology (Part 5).

Four real investigations, recorded end to end

The two cases above are composites. These four are not — they are unedited screen recordings of the workbench running against a live Azure tenant, captured in one sitting. Each clip starts on an empty chat and runs the whole way through: scoping to a workload, switching the turn to Deep investigation, typing the question, the war room assembling, All Hands On Deck, the launch, the specialists working in parallel, and the written verdict at the end. Nothing is sped up beyond playback rate and nothing is cut.

You've already seen the first one — the 5xx run at the top of this post. Here's what it actually concluded, and then three more.

1. "One in five requests is failing and nothing was deployed"

The question in the title of this post, asked verbatim: roughly one in five requests to the azsupagent app fails with a 5xx while other users see no problem at all; nothing was deployed.

The verdict is the interesting part, because the answer is nothing is broken. There is no server-side fault at all — the 5xx responses are generated at the Container Apps ingress, in front of the app, because the app is pinned to a single replica (min = max = 1, zero restarts all day). Concurrent requests beyond that one instance's capacity get 502/503/504 while requests already in flight on the same replica finish normally — which is exactly the "one in five fails, the rest are fine at the same moment" shape. Crucially, the loud suspects are cleared with evidence, not skipped: no crash loop (replica count flat at 1.00, restart count 0), no autoscale churn, no Postgres exhaustion (8.88 active connections, 0 failed, CPU 8.7%), no storage dependency (100% availability across 24h).

(Recording at the top of this post.)

2. "It's slow at peak but the database dashboards look fine"

Different symptom, different specialists leading, same underlying cause as the run above — which is a good demonstration that the conclusion follows the evidence rather than the phrasing of the question. Replica count is pinned at 1.00/1.00 across the full seven-day window, so at peak the requests queue in the ingress buffer. That queueing adds seconds of latency that never appears as CPU, memory or database pressure, which is precisely why the DB dashboards look clean and why nobody can reproduce it off-peak.

Note that the database is actively disproven rather than merely unexamined: disk_queue_depth is 0 in every one of 96 buckets, peak IOPS is 4 out of a provisioned 240, and active connections peak at 15 with zero failures. That distinction — invalidated vs. not checked — is the whole argument for the hypothesis tree.

3. "What changed in the last 48 hours, and could any of it hurt me?"

This one shows the War Room used as a change-risk tool rather than an incident tool, and it found something more uncomfortable than a bad deployment. The riskiest change in the window was not made by a human at all: a service principal holding Contributor at the tenant root management group created a new, undocumented action group. The object it created is benign in content — email and SMS only, no webhook — so alerting was never broken. The identity is the finding: an unowned, unattributed non-human principal with tenant-wide write authority and root-scope policy rights.

The secondary finding is independent and equally unglamorous: shared storage account keys were retrieved three times in the small hours on an account that still permits shared-key auth, accepts TLS 1.0, and has no network restrictions at all. Meanwhile the five human container app deployments in the same window are explicitly cleared of causing instability.

4. "No incident — just tell me what breaks first"

The last one is the proactive case, and it's the one I'd actually run on a Friday afternoon. No symptom, no page, no incident: just audit this workload end to end and tell me what fails first, how bad the blast radius is, and the single highest-value fix this week.

The answer is the PostgreSQL flexible server, and the audit is specific about why: HA disabled, a single primary pinned to one availability zone, zero read replicas, geo-redundant backup disabled, seven-day retention, and storage auto-grow off. A zone outage — or an ordinary planned maintenance event — takes it fully offline with no failover target and nothing to geo-restore from, so the blast radius is a 100% workload outage with recovery bounded by manual restore time. It also flags something no single-resource check would catch: the database is the only resource in one region while the entire rest of the workload sits in another, so every data path already leaves the app's region for zero resilience benefit.

Scoped to the workload, scoped to the tenant

A War Room investigation doesn't have to boil the ocean. Point it at a Workload and it's constrained to that application's resources — its subscriptions, resource groups and resource IDs — so it reasons about the thing you care about instead of wandering the whole estate. And because the Azure Support Agent is multi-tenant aware, every investigation carries its tenant and connection context, so each tool call targets the right subscription in the right directory. One operator can run investigations across several customers' tenants from the same workbench without the signals ever crossing.

You don't have to start in the chat

Most investigations don't begin with someone typing a question — they begin somewhere else in the workbench noticing something is off. So the War Room is wired to be handed off to. When a proactive feature surfaces a risk, it can open a Deep investigation pre-loaded with the right workload and a ready-made prompt:

Backup/DR coverage finds a resource with no protection → "investigate this gap."
Change Explorer flags a risky change → "investigate this change and its blast radius."
Performance Profiler spots a metric pressing against its Azure Monitor threshold → "why is this the binding bottleneck?"
Retirement Radar finds a workload in the blast radius of an upcoming Azure retirement → "investigate this impacted workload."
Telemetry Intelligence sees a failure spike on an operation → "investigate this spike."
Architecture Memory opens the War Room with that architecture's memory already attached.
Identity findings and workload cards stage a scoped chat the same way — the workload card going straight into a deep reliability review.

Each one hands the War Room its context on a plate: deep mode on, the right workload selected, the prompt already written, the recommended bench loaded. What it deliberately does not do is start investigating. Every hand-off stops at the specialist-selection gate — nothing is dispatched until you click Launch. A dashboard can stage an investigation; only a person starts one.

There's a durable twist worth stealing, too. A browser hand-off is one-shot — it dies on a refresh — so as the War Room opens it mirrors that context into a server-side Case File, reusing the workload's already-open case (appending a note) instead of piling up duplicates. The investigation is a chat; the case is the record that outlives it.

A hand-off arrives fully staged — deep mode on, workload scoped, prompt written, bench loaded:

And then it stops, deliberately, at the gate and waits for a human:

From one workload to the whole fleet

The same machinery scales past a single incident. Select a handful of workloads in the fleet view, hit Deep review, and the workbench spins up one investigation per workload — its own chat, titled for the workload, the full eight-specialist bench, and a shared reliability brief: review this workload across networking, identity, compute, storage, security, monitoring and cost, and name the top risk. A whole portfolio gets swept for latent risk in one pass instead of you babysitting workloads one at a time.

It's throttled on purpose. A batch is capped, and the launches go out through a small concurrency gate rather than all at once — twenty simultaneous eight-specialist investigations would stampede any model provider's rate limits, and an ops tool that DDoSes your own AI quota isn't much of an ops tool. Each workload comes back with its own tree, root cause and confidence, so you can triage the fleet by what the evidence actually says. It's the difference between reacting to the workload that paged you and proactively asking all of them "is anything wrong?" — and it's the bridge to Mission Control and scheduled automations later in the series (Part 5).

It starts where you'd expect — the fleet view, with the workloads you care about selected:

What comes out: severity, confidence, and next steps

An investigation isn't just a wall of text — it resolves into a structured result you can route and reuse: a stated root cause and summary, a severity (info, warning, error or critical), the evidence chain behind the verdict, and a ranked list of next actions. Both lists are capped, so what you get is a shortlist to act on rather than a transcript to wade through. On top sits the confidence score from earlier — three parts strongest-hypothesis, one part whole-slate — so three inconclusive maybes score low and look exactly as shaky as they are. Because the shape is structured, the same result feeds a Case File, a ticket, or the next investigation without anyone re-typing it.

Those next actions can include the precise commands to fix the problem — but this is still the Converse pillar, so they stay proposal-first. The War Room tells you what it would do and why; actually doing it is the approval-gated territory of the Act pillar (Part 5). Nothing runs behind your back.

Grounded, not guessed

It's worth saying plainly why this is trustworthy, because "AI incident response" can mean a lot of hand-wavy things. Every phase here runs against live Azure evidence through MCP tools that are read-only by default — the Azure Support Agent is investigating, not changing anything. Each tool call and its result is captured in the timeline, so the conclusion is backed by a visible trail of exactly which resources were queried and what they returned. Hypotheses that the evidence contradicts are marked invalidated and kept in the tree rather than quietly dropped — the record shows the roads not taken. That read-only discipline holds even for the heavier tools: the performance profiler reads, the sandbox-VM checks read, and any write-classified tool is filtered out of the toolset the moment Deep mode starts — an investigation is structurally incapable of mutating your estate. This is the same grounding discipline the whole product is built on, and Part 6 is devoted to it in full.

It gets smarter about your estate over time

There's one more piece that turns a good investigation into an institutional one. Each architecture you've mapped can carry an Architecture Memory — the intended design, the security model, known gaps, diagnostic hints — and that context is injected into every phase of an investigation as expert background. So the War Room doesn't just know generic Azure; it knows this workload's quirks.

And the loop closes: when an investigation nails a root cause, you can save that RCA back into the architecture's Memory with a click. The next time something goes sideways on the same workload, the agent recalls what happened last time. Your incidents stop being disposable and start compounding into knowledge — the tribal wisdom that usually lives in one senior engineer's head, captured where the whole team can reuse it.

And a saved RCA isn't a dead note. It carries its confidence, it's de-duplicated against the exact investigation it came from (save it twice and you still have one entry), and it lands in the memory's known-issues section — the section that gets injected first when context has to be trimmed to fit. Your estate quietly accumulates its own case law, and the most expensively-learned facts are the last ones to fall out of the prompt. When you want a finding to leave the workbench entirely, one click escalates an investigation into a tracked Case that back-references the exact chat and message. And every Deep investigation you've run is kept in a searchable history, each with a digest — root cause, hypothesis counts, confidence — and its full, pinnable tree one click away.

The whole learning loop is one button at the bottom of a finished investigation:

…and the finding reappears in the architecture's known-issues section, ready to be injected into the next investigation on that workload:

Point it at almost anything

Root-causing a 5xx is the headline, but the same parallel-specialist machinery answers a much wider range of questions. A sampling of what people point the War Room at:

Proactive audits — "check every NSG for 0.0.0.0/0 rules" (Security & Exposure).
Change correlation — "did last night's change cause this?" (Monitoring & Logs).
Retirement impact — "which workloads does this upcoming retirement hit?" (Reliability).
Cost spikes — "why did this subscription jump 30% this month?" (Cost & Governance).
Performance — "what's the binding bottleneck on this workload?" (Reliability & Performance).
Security posture — "is any storage account publicly reachable?" (Security & Exposure).
Connectivity — "why can't this subnet reach that private endpoint?" (Networking).
Governance — "is everything tagged to our taxonomy?" (Cost & Governance).

Same four phases, same read-only discipline, same navigable tree — pointed at whatever question you have this week.

How it's judged

Every post in this series ends with the same honesty check — how you'd know the feature is actually doing its job, not just demoing well:

Every answer shows its work. Each turn carries a visible reasoning + tool-call timeline; the conclusion is traceable to the exact Azure queries behind it.
Hypotheses are validated against live evidence. The tree records what was checked, what was confirmed, and what was ruled out — each verdict tied to specific evidence.
Confidence is earned, not asserted. The score is computed from how cleanly the hypotheses resolved — weighted toward the strongest one — so a shaky conclusion looks shaky.
Read-only by default. The whole investigation runs against your estate without changing a thing; any write stays proposal-first and approval-gated.
Nothing launches itself. Dashboards can stage an investigation with full context, but specialists are only ever dispatched when a human clicks Launch.
Turns are durable. Cancel a turn or close the tab and the work continues server-side and is saved — evidence and all — so nothing is lost.
You choose the bench, or the AI suggests it. A focused roster means less noise and lower cost without losing parallel coverage.
Bounded by design. Published caps on iterations, wall-clock time, tree size and parallelism force convergence, so it lands an answer instead of wandering — and can't quietly run up a bill.
Real diagnostics, still read-only. Performance profiles, access lookups and in-guest checks all run without the power to change anything.
Reusable output. Every investigation resolves to severity, confidence and next actions — escalatable to a Case and recalled on the next incident.

Try it yourself

If you deployed the Azure Support Agent after Part 1, open a chat, ask it something real about your estate, and watch the timeline build. Then toggle Deep mode on a genuinely thorny question and let the War Room run — the hypothesis tree is the single best way to feel what an AI ops workbench does that a chatbot can't. No live tenant yet? Demo mode runs the same experience on synthetic data, with nothing to risk.

Next time — Part 3: Map your estate — from proliferation to a living picture. The War Room is only as sharp as its understanding of what's deployed, so next we turn resource proliferation into something you can reason about: Workloads with explainable health scores, Autopilot that proposes them for you, live architecture diagrams reverse-engineered from what's actually running, the Architecture Memory you just saw feeding investigations, and the whole-tenant Estate Graph. Mapping is where "proactive" gets its eyes.

It's open source and free to run. Three ways to start, in rising order of commitment:

Explore demo mode — synthetic data, no connection required, nothing to risk.
Deploy to your tenant — one click to Azure Container Apps; private networking; your data never leaves.

Or main repository https://github.com/zmustafa/AzureSupportAgent

Part 2 Original

Proactive Resiliency with Azure App Service Auto Heal

lakprasad — Wed, 22 Jul 2026 20:52:05 GMT

Modern cloud applications are expected to remain resilient - they must observe, protect, recover, and preserve service continuity when workloads begin to behave unpredictably. This includes scenarios such as transient faults, memory pressure, dependency timeouts, or sudden spikes in failed requests. While traditional monitoring detects these issues, recovery often depends on manual intervention, increasing Mean Time to Recovery (MTTR) and operational overhead.

In Azure App Service, Auto Heal is one of the platform’s bulit-in resiliency controls. It is designed to detect unhealthy application behavior and automatically take corrective action without human intervention.

This makes Auto Heal less of an operational convenience and more of an infrastructure self-preservation capability aligned with the Azure Well-Architected Framework, helping applications to recover gracefully while preserving availability and diagnostic context.

Why Auto-Heal Matters and Why Infrastructure Needs Self-Preservation

In mission-critical environments, infrastructure cannot depend entirely on humans noticing every failure pattern in real time. Applications can degrade gradually, dependencies can slow down, memory can grow unexpectedly, and individual instances can enter unhealthy states while the rest of the platform continues operating.

Without automated mitigation, these symptoms often become operational events:

A process consumes excessive memory.

Requests begin taking longer than expected.

HTTP failures increase.

One instance starts affecting user experience while others remain healthy.

Auto Heal allows App Service to respond to these conditions using pre-defined rules and it recognize that the workload has crossed a boundary where recovery action is safer than continued degradation. This way it helps maintain application stability in production environments.

How Auto-Heal Works

Trigger Conditions (Detecting Unhealthy State)

Auto-Heal monitors application behavior using configurable signals such as:

Trigger	Infrastructure signal
Request duration	The app is responding too slowly or threads may be blocked.
Request count	The app should recycle after a controlled amount of traffic.
HTTP status codes	The app is returning repeated failures.
Memory limit	The process or container is approaching unsafe resource consumption.

These conditions are evaluated over a defined time window and threshold, enabling precise detection of abnormal patterns.

Mitigation Actions (Self-Recovery)

When a trigger condition is met, Auto-Heal executes one or more corrective actions:

Recycle worker process (most common)
Restarts the application process for the affected instance.

Capture memory or crash dumps
Preserves diagnostic data for post-incident analysis.

Log events
Records Auto-Heal activity for auditing and troubleshooting.

Execute custom actions (Windows / supported Linux scenarios)
Run scripts or executables for advanced remediation.

Each action is applied per instance, ensuring that other healthy instances continue serving traffic.

Controlled Recovery Behavior

Auto-Heal works instance-by-instance, avoiding full application downtime in scaled environments.

When combined with Health Check, unhealthy instances are temporarily removed from load balancing before recovery.

A configurable startup override window prevents false triggers during application initialization.

Key Benefits:

Improved application availability
Automatically recovers from transient failures such as HTTP 5xx bursts, slow requests, or memory pressure.

Reduced Mean Time to Recovery (MTTR)
Eliminates dependency on manual restarts by triggering corrective actions instantly.

Built-in diagnostics at the point of failure
Captures logs, memory dumps, and telemetry during failure - before state is lost.

Operational efficiency
Reduces the need for continuous monitoring and manual intervention, especially during off-hours.

Seamless integration with platform capabilities
Works alongside Health Check, Azure Monitor, and Application Insights for a comprehensive resiliency strategy.

Importantly, Auto-Heal is included as part of Azure App Service and does not incur additional cost beyond standard logging and telemetry storage.

However, Auto Heal should be viewed as a guardrail, not a cure. It does not eliminate the root cause of a memory leak, dependency bottleneck, bad release, or application deadlock. Instead, it helps preserve runtime health by taking controlled action when the application reaches an unhealthy state. A resilient platform does not assume failures will never happen. It assumes failures will happen and provides mechanisms to contain, recover, and learn from them.

Hence, validate the following while configuring from a resiliency perspective:

Validation question	Why it matters
Are triggers mapped to real failure patterns?	Avoids rules that are too generic or irrelevant.
Are thresholds based on production baselines?	Prevents unnecessary recycling or delayed recovery.
Is startup time accounted for?	Avoids triggering Auto Heal during normal warm-up.
Is the action safe for the workload?	Prevents loss of in-memory state or avoidable user impact.
Are logs and diagnostics enabled?	Ensures recovery events can be investigated afterward.

Configuring Auto-Heal

Auto-Heal can be configured using multiple approaches:

Azure Portal

- Open the App Service resource in the Azure portal.

- Go to Diagnose and solve problems.

- Select Diagnostic Tools > Open Auto-Heal.

- Configure triggers, thresholds, startup delay, and mitigation actions.

Application configuration (web.config) (Windows)
Enables rule definition alongside application code

Infrastructure as Code (IaC)
ARM, Bicep, CLI, or SDK-based configuration for consistency across environments

This flexibility allows teams to standardize Auto-Heal policies across workloads.

Validating Auto-Heal in Action

Operational visibility is critical. Azure provides multiple ways to validate that Auto-Heal has triggered and executed correctly.

Azure Portal – Auto-Heal History

Navigate to:

Diagnose and Solve Problems → Auto-Heal → History

This view provides:

- Trigger conditions met

- Timestamp of execution

- Actions taken

- Status and outcome

Application Event Logs

Accessible via:

- Portal: Diagnostic Tools → Application Event Logs

- Kudu:
  LogFiles/EventLog.xml

Look for Auto-Heal related events indicating:

- Trigger activation

- Process recycles

- Success or failure of actions

Kudu Console (Advanced Diagnostics)

Kudu provides deep runtime visibility:

- Process Explorer
  Observe worker process (w3wp) restart via Process ID (PID) changes

- Log Stream
  View real-time errors leading up to Auto-Heal execution

- File system access
  Retrieve memory dumps and logs from: D:\home\LogFiles\

Review Web App Restarted & HTTP Setting Errors

- In App Service, Diagnose and Solve Problems, search for Web App Restarted, Filter by time and select Auto-Heal to view status → it will show status same as Auto Heal History

- For HTTP error scenarios, check HTTP Setting Errors for HTTPS unavailability and related triggers.

Azure Monitor Metrics

Track Auto-Heal activity using:

- WorkerProcessRecycleCount: Indicates how often instances are recycled
- HealthCheckStatus: Shows instance recovery lifecycle

These metrics can be used to:

- Build dashboards

- Configure alerts

- Correlate with performance degradation

Application Insights Integration

When enabled, Application Insights helps:

- Correlate Auto-Heal events with exceptions and dependencies

- Analyze performance trends leading to failures

- Validate recovery timelines

Best Practices for Effective Auto-Heal Usage

To maximize effectiveness:

Define safe and practical triggers

- Start with conservative thresholds to avoid unnecessary restarts.

- Continuously refine rules using diagnostics and production behavior to improve accuracy over time.

Pair Auto-Heal with platform resiliency features

- Use Health Check so traffic is routed away from unhealthy instances during recovery.

- Use autoscaling alongside Auto-Heal: Auto-Heal addresses runtime instability, while auto scale responds to load.

Monitor and learn from recovery patterns

- Track frequent events as repeated activations may indicate deeper application issues.

- Enable observability through Azure Monitor and Application Insights for end-to-end visibility.

Key Considerations and Limitations

While powerful, Auto-Heal must be used thoughtfully:

Auto Heal is not a substitute for root cause fixes
Repeated triggers indicate underlying issues that must be addressed

Overly aggressive rules can cause churn
Frequent recycles may impact user experience

Platform differences (Windows vs Linux)

- Memory-based triggers are not supported on Linux

- Custom actions have limited support on Linux

- Windows provides the most comprehensive feature set

Single-instance applications experience downtime during recycle
Scaling to multiple instances is recommended for production workloads

Dependency failures are not resolved by Auto-Heal
Recycling does not fix external service outages. The goal is not to restart applications more often. The goal is to make recovery safer, faster, and more intentional when unhealthy runtime states occur. The dependency failures need to be investigated further.

Bringing It All Together

Auto-Heal is a foundational capability for building resilient, self-recovering applications on Azure App Service. When combined with Health Check, monitoring, and scaling strategies, it enables:

Faster recovery from runtime faults

Improved availability and user experience

Reduced operational burden on engineering teams

By thoughtfully configuring triggers, validating behavior through diagnostics, and continuously refining rules, organizations can transform Auto-Heal from a simple restart mechanism into a strategic resiliency control.

Azure Support Agent — Part 1: Building an AI Azure Ops Workbench

zmustafa — Mon, 03 Aug 2026 13:11:39 GMT

Picture a busy afternoon: an app starts throwing 5xx, and you go looking for why. The good news is that Azure almost certainly already holds the answer. The Portal has the resource. Azure Resource Graph knows what's actually deployed. Monitor has the metrics, Log Analytics has the logs, Advisor has the recommendations, and the CLI reaches the rest. Azure is remarkably data-rich — very little about your estate is truly hidden. The real challenge at scale is human bandwidth: stitching a dozen surfaces into a single answer while the clock is running.

There's a second, quieter category of work, too — the things that never prompt you to open a tab in the first place. A database whose backups quietly stopped succeeding. A workload that grew business-critical before anyone gave it a DR pair. A Key Vault secret ticking toward expiry. The signals to catch every one of these already exist in Azure; what's missing is simply someone (or something) with the time to watch all of them, continuously, across the whole estate.

Now multiply that across a real environment — hundreds of resources, dozens of teams, and a platform that keeps evolving as Azure ships new capabilities and retires older ones. Nobody here is short on skill or data; there's just far more signal than any human can watch at once. And that — always-on, correlate-everything, catch-it-early work — is exactly what AI is good at. It's the gap this workbench is built to fill.

What if your subscription had a copilot that already knew the answer — and had been watching for the problem before you asked?

Meet the Azure Support Agent

The Azure Support Agent is an open-source, AI-driven operations workbench that you deploy into your own tenant. Point it at a subscription and it discovers your workloads (e.g. find related resources using AI inferencing), reverse-engineers live architecture diagrams (how difficult has it been for you to catch-up with diagrams? how about live+living documentation?), runs Well-Architected-style assessments, and puts a team of specialist AI agents at your disposal to investigate, monitor, and remediate.

It talks to Azure through the official Azure MCP server and a Microsoft Graph (Entra ID) MCP server — the same Model Context Protocol that's quickly becoming the standard way for AI to use real tools. Between them that's over a hundred read-classified tools, so the agent reasons over live evidence from your estate, not a training-data guess about how Azure works. In other words, it doesn't replace any of Azure's tooling — it orchestrates it on your behalf, on top of the platform APIs you already trust.

And it runs where your data already lives. One click deploys it to Azure Container Apps in your subscription; nothing about your estate leaves your tenant.

the Architecture Designer showing reverse-engineered architecture diagram of your workload with the help of AI, overlay with assessments, launch tools direclty

Converse — a chat that reads your tenant, and a war room for the hard ones (Part 2)

Back to that bad afternoon — except this time you don't open eight tabs. You ask, in plain English: "Why is the checkout API throwing 5xx?" — and the agent answers by doing the work, not by guessing from training data. It queries your live estate through the MCP tools and streams its reasoning and every tool call back to you as it goes, so you can see exactly which resources it looked at. Those turns are durable: reload the page and the timeline is still there, and if you cancel a long answer the work keeps running server-side instead of evaporating. You can keep several investigations going at once, paste in a screenshot of a failing blade, and lean on a starter suggestion when you're not sure what to ask.

When a question is genuinely hard, you switch on Deep mode and a War Room convenes. Instead of one model guessing, a team of specialists goes to work in parallel — a Networking specialist checks NSGs, private endpoints and DNS; an Identity specialist looks at managed identities and expiring secrets; Compute, Storage, Security and Reliability specialists each take their own angle — every one of them pulling live evidence from your estate at the same time. They form hypotheses, throw out the ones the evidence contradicts, and converge. Ninety seconds later you have a conclusion rather than a hunch: the app's Key Vault secret expired overnight; here's the dependency chain that proves it, and here's the exact command to rotate it. The whole investigation is saved as a navigable hypothesis tree attached to the chat, so the next person doesn't start from zero.

Map — turn sprawl into a picture you can actually use (Part 3)

You can't reason about resources you can't see, and almost nobody has a current map of their own estate — so the agent builds one for you. It groups raw resources into Workloads (the resources that actually make up one application) and gives each a composite, explainable health score; if you'd rather not draw those boundaries by hand, Autopilot studies the estate and proposes the groupings for you to approve.

Then it gets visual. Point it at a workload and it reverse-engineers an interactive architecture diagram from what's genuinely deployed, inferring the real relationships — NIC to subnet to VNet, app to plan, private endpoint to target — instead of trusting a stale Visio. It keeps a persistent Architecture Memory for each design (intended topology, security model, resiliency targets, known gaps) that later feeds those War Room investigations as expert context, and it flags drift when the live estate wanders from the design. It will even draft the on-call "know-me" doc for a workload, auto-filling what Azure knows and marking human-only facts — SLAs, escalation paths, RTO/RPO — as explicit TODOs rather than inventing them. Zoom all the way out and the Estate Graph renders your whole tenant as one searchable, workload-aware map, with cost, retirement and RBAC overlays and deep links straight to the underlying resource.

Use AI to discover your Azure workloads from a management group/subscription/rg scopes, then group their resources to perform proactive operations

A reverse-engineered estate graph of your Azure Workload

Assess — go looking for trouble before it finds you (Part 4)

This is where "proactive" stops being a slogan. Most tooling waits to be asked; the Azure Support Agent runs a battery of assessments that hunt for risk before the ticket exists, each judging your estate against a reviewed, versioned baseline rather than a vibe.

It starts with coverage: which resources are missing the Azure Monitor baseline alerts they ought to have, and which are shipping no diagnostic logs at all — then it hands you ready-to-apply Bicep or Terraform to close the gaps. It audits backup and DR the same way: is each resource actually backed up, with adequate retention, an offsite copy and a DR pair that's been drilled — or will you find out during the outage? A Performance Profiler reads live metrics and ranks every resource by saturation on a heatmap, naming the workload's binding bottleneck before your users feel it. And for the big picture it scores a workload against the five Well-Architected pillars with reproducible, deterministic math (never a random number a model made up), turns the architecture into a quantified FMEA risk register, and gives you a read-only identity and RBAC posture — expiring credentials, ownerless apps, MFA gaps, and who can really do what. The payoff is a change of timing: you stop discovering that a database was never backed up at the worst possible moment, and start seeing it on a dashboard on a quiet Tuesday.

Azure Monitor Baseline Alerts (AMBA) assessment against your Azure Workload - showing where you have/have not configured alerting

Performance Profiler - unified matrix view of all your resources in an Azure Workload and their performance metrics at a point in time

AI driven change explorer on what changed in your Azure Workload during a time window

Conduct AI assisted Failure Mode and Effects Analysis (FMEA) against your Azure Workload

Act — don't just find it, fix it (safely) (Part 5)

Findings that die in a dashboard help no one, so the last pillar is about closing the loop — carefully. You can dry-run an Azure Policy deny and measure its real blast radius — exactly which resources it would hit — before you enforce a thing. You can straighten out ownership and tagging from a single owner directory and tag census, applying corrections as real tags with snapshots and a one-click revert if anything looks off. And Change Explorer tells you, in plain English, what changed across your estate, who changed it, and whether it was risky — the forensics you always wish you had mid-incident.

From there it helps you get ahead of the calendar and, eventually, take yourself out of the loop. Retirement Radar maps Azure service retirements and breaking changes to the specific workloads they'll hit, with real deadlines; the Reservations view tells you whether your commitments are actually paying off or quietly leaking money. Mission Control runs every analysis on a workload at once for a single go/no-go verdict — which you can then put on a schedule with Automations, routing results to Teams, Jira or ServiceNow so the whole loop runs without you babysitting it. Everything that writes stays proposal-first: the AI suggests, you approve.

Mission Control's per-workload go/no-go verdict, pick a workload, launch and let it warm up

Unified view of Azure retirements and breaking changes against your Azure Workload

Built to be trusted

None of this is worth much if you can't hand it the keys with a clear conscience, so the trust model is deliberately conservative. The Azure Support Agent is read-only by default — every write is approval-gated and recorded in a full audit log, that can be sent to Splunk, CrowdStrike NG SIEM, or others. Full EntraID based authentication, It runs in your Azure Container App with private endpoints intra-service, so nothing about your estate ever leaves your tenant. And the AI is off until you switch it on: bring your own provider — OpenAI, Azure OpenAI, Anthropic Claude, Google Gemini, GitHub Copilot, Mistral, or a local model via Ollama or LM Studio — and there is zero LLM traffic until you explicitly configure one. This is the difference between a chatbot that answers from memory and a workbench that uses tools against live evidence, shows its working, and leaves you with artifacts you keep. It matters enough that a whole later post is devoted to it — Part 6 — covering the full access model and how the answers stay grounded, reproducible and free of fabrication.

Where this series goes

This is Part 1 of a seven-part tour. Each post ahead goes deep on one pillar of the Azure Support Agent, leads with a real screenshot, and stands on its own — so skip to whichever one solves your problem this week:

Part 2 — The War Room: watch a team of AI specialists root-cause a 5xx in ninety seconds.
Part 3 — Map your estate: live architecture diagrams, Workloads, know-me docs, the Estate Graph.
Part 4 — Assess before it pages you: alert/telemetry and backup/DR coverage, Well-Architected scores, FMEA risk registers and identity & RBAC — with the IaC to fix the gaps.
Part 5 — Act: govern, fix, and stay ahead: Policy dry-runs and blast radius, ownership & tags, change forensics, retirement radar, reservations and Mission Control.
Part 6 — Letting AI into your Azure: the read-only-by-default access model, grounded and deterministic answers, and bring-your-own-model.
Part 7 — How it's wired: the agents, the MCP layer, and one-click deploy — for the builders.

Whether you're on-call chasing a root cause, an architect who needs a current diagram, on the security team watching RBAC and secrets, or a FinOps owner hunting waste, there's a part above with your name on it.

The bottom line

The promise of the Azure Support Agent is a shift in posture. Your estate stops being a mystery you interrogate after something breaks, and becomes a system that tells you what's wrong — often before anyone gets paged.

Picture next Monday. The expiring Key Vault secret shows up on a dashboard the Thursday before, and gets rotated in the quiet — so the 5xx afternoon simply never happens. The database with the failing backups is flagged, with generated Bicep to fix it, a week ahead of the quarterly DR drill instead of during the post-mortem. Nothing about your Azure changed except when you found out.

Try it yourself

The Azure Support Agent is open source and free to run. Three ways to start, in rising order of commitment:

Explore demo mode — synthetic data, no connection required, nothing to risk.
Deploy to your tenant — one click to Azure Container Apps; private networking; your data never leaves.

Or main repository https://github.com/zmustafa/AzureSupportAgent

Next time — Part 2: The War Room — watch AI root-cause a 5xx in ninety seconds. We'll go inside a live Deep-mode investigation: how a team of specialist agents fans out across your estate in parallel, forms and discards hypotheses against real evidence, and converges on a root cause you can navigate — and trust. It's the single best demonstration of what an AI ops workbench actually feels like.

Part 1 Original

Kernels Behind the Wall: Custom Python 3.10-3.14 Jupyter Kernels for Air-Gapped Azure ML

Sabyasachi-Samaddar — Tue, 07 Jul 2026 18:28:36 GMT

Purpose

Regulated organizations - banks, insurers, healthcare providers - increasingly run Azure Machine Learning on compute instances that have no outbound internet access. Their data scientists still need modern Python versions and a rich, curated set of libraries. This document exists to close that gap. It captures a repeatable, field-tested pattern for delivering fully loaded Python 3.10-3.14 Jupyter kernels to an air-gapped Azure ML compute - without ever running a package resolver on the locked-down side. The technique itself is version-agnostic: the build, pack, ship, and unpack mechanism works for any Python version that conda can create. This guide uses Python 3.10-3.14 only as a representative range - 3.10 to match the built-in Azure ML kernel and 3.14 as today's ceiling - but the same steps apply equally to 3.9, a future 3.15, or any other version. The only version-dependent part is the package list itself (for example, the Azure ML SDK v1 tops out at Python 3.11).

What This Document Is

A hands-on runbook for building, packaging, shipping, and installing custom Python 3.10-3.14 Jupyter kernels on an air-gapped Azure ML compute instance.
A build-once, ship-an-artifact pattern that uses conda-pack, an Azure Storage blob reached over a private endpoint, and the compute's managed identity.
A cleaned, ready-to-install package list - tuned per Python version - in which the known traps from the original Azure ML package export have already been fixed (conda internals, a CLI masquerading as a library, and standard-library-shadowing backports removed; wrong package names and duplicates corrected).
A troubleshooting reference for the failures that actually happen: RBAC 403s, the azureml v1 Python ceiling, architecture and glibc mismatches, and conda-pack gotchas.

What This Document Is Not

Not an introduction to Jupyter, conda, or Python packaging fundamentals.
Not a guide to provisioning Azure ML workspaces, virtual networks, or private endpoints - those are assumed to already exist.
Not for internet-connected computes, where a normal pip install or conda environment is all you need.
Not a Windows or ARM recipe - the build box must be Linux x86_64 that matches the target compute.
Not an official Microsoft product manual; it is a practitioner's field guide.

Executive Summary

An air-gapped Azure ML compute presents two problems at the same time: it cannot reach PyPI or conda, and it ships with a single built-in Python 3.10 kernel. This guide solves both with one idea - build the environment where internet exists, freeze it into a single self-contained artifact, and ship that artifact across the air gap. The same mechanism works for any Python version conda can build - the 3.10-3.14 range used throughout is only a worked example, not a limit.

Build once on an internet-connected Linux x86_64 box, then conda-pack the environment into a single relocatable tarball.
Upload the tarball to a blob container the compute can reach over a private endpoint; on the compute, install_offline_kernel.py uses the managed identity to download and unpack it - with zero package resolution on the locked-down side.
Clean the raw package list first: strip conda internals, a CLI masquerading as a library, and backports that shadow the standard library.
Keep the azureml SDK v1 for Python 3.10 and 3.11; drop it for 3.12-3.14 and use azure-ai-ml (v2) instead.
Blob upload and download need data-plane roles (Storage Blob Data Contributor or Reader) - control-plane Owner or Contributor is not enough.
Custom kernels persist across compute stop/start; only deleting the compute removes them.

Here is a fun situation I keep running into at regulated customers. A data scientist opens a brand-new Azure ML compute instance inside a locked-down bank network, types pip install pandas, and gets… nothing. No wheels. No index. No error worth reading. Just a cursor blinking back with the quiet confidence of a machine that has never once touched the internet and does not intend to start now.

That is the environment we are dealing with: an air-gapped compute. And air-gapping is only the first of two problems. The second is that Azure ML Studio ships exactly one built-in kernel — Python 3.10. If you need 3.11, 3.12, 3.13, or 3.14, or you want that 3.10 kernel preloaded with your organisation’s curated data-science stack instead of the bare default, the only supported path is to register your own Jupyter kernel. On a normal box you’d pip your way there in a coffee break. Here you can’t, for the reason above.

This post is the pattern I use to solve both at once: build the environment somewhere that does have internet, freeze it into a single self-contained artifact with conda-pack, ship that artifact to a Storage blob the compute can reach over a private endpoint, and unpack it in place. No dependency resolution on the secure side. No firewall exceptions. No six-week change request that ends in “no.”

The short version

Two constraints, one technique. Air-gapped compute (no PyPI/conda) and a single built-in 3.10 kernel. Custom kernels solve the version/packages gap; conda-pack solves the no-internet gap.
Build once, ship a tarball. Create the env on an internet-connected Linux x86_64 box, conda-pack it, upload to blob, download + unpack on the compute. Zero resolver runs on the locked-down side.
Don’t trust the raw package list. A curated AML image’s package dump contains conda internals, a CLI masquerading as a library, and two backports that shadow the standard library. Clean it first (Section 4).
azureml SDK v1 caps you at Python 3.11. Keep azureml-* for 3.10/3.11; drop it on 3.12–3.14 and use azure-ai-ml (v2).
“Owner” does not grant blob I/O. Uploads/downloads need a data-plane role (Storage Blob Data Contributor / Reader), not control-plane Owner or Contributor.
Kernels survive stop/start. They live on the persistent user disk; only deleting the compute wipes them.

Before you start

Five things need to be true before any of the commands below will work. Confirm them now, not at Section 6:

A Linux x86_64 build box with internet — Ubuntu 22.04 or similar. Must be x86_64 (see the arch note in Section 3) and ideally a similar/older OS than the compute (see the glibc note there too).
An Azure Storage account reachable from the compute — over a private endpoint from the compute’s VNet, with a container to hold the tarball.
A managed identity on the compute instance — system- or user-assigned, attached to the compute (not just the workspace), with Storage Blob Data Reader on that account.
Rights to assign RBAC — Owner or User Access Administrator on the storage account, so you can grant the data-plane roles in Section 6. If you don’t have this, line up whoever does before you begin.
Terminal access to the compute — via AML Studio, where you’ll run the installer as azureuser.

1. Why the obvious approaches don’t work

Before the solution, it’s worth being honest about why the easy options fail, because someone on the call will suggest all of them:

The tempting shortcut	Why it doesn’t survive contact with an air-gapped compute
pip install -r requirements.txt	No route to PyPI. DNS or connect timeout. There is nothing to resolve against.
conda create -n env python=3.12 …	Can’t reach conda channels, and now also blocks on the Anaconda ToS endpoint it can’t contact.
Just use the built-in kernel	It’s Python 3.10 only, with the stock package set. No version choice, no curated stack.
“Can we open the firewall briefly?”	In a regulated tenant, that’s a security review measured in weeks and usually ends in a polite no.
Copy a wheelhouse folder by hand	Fine for pure-Python, but collapses the moment a package needs a compiled binary the target can’t build.

The insight: stop trying to make the compute reach out. Build everything outside, freeze it into one self-contained artifact — interpreter, packages, and native .so files included — and hand it to the compute through the one channel it already trusts: its own VNet’s private endpoint to Azure Storage.

2. The approach: build, pack, ship, unpack

Four steps. Assemble the env on a machine with internet, compress it with conda-pack into one .tar.gz, upload it to a blob container, and run a small installer on the compute that downloads, unpacks, and registers the kernel.

Why conda-pack instead of rebuilding from a pip freeze on the target? Because a freeze from a curated AML image is not a clean, reinstallable manifest. That env was assembled by conda plus internal feeds, so its pins are internally contradictory — they will not resolve on a fresh Python. conda-pack sidesteps all of it by shipping the exact binaries that already work. The target never runs a solver, so there is nothing left to conflict.

📌 Note

A conda-pack tarball is a frozen snapshot of a working env — same site-packages, same shared objects, same interpreter. Once conda-unpack rewrites the embedded absolute paths, it runs exactly as if it had been built on the compute. No network, no channels, no ToS prompt.

3. Part 1 — Build the environment (Linux x86_64)

Where: an internet-connected Linux x86_64 box (Ubuntu 22.04 or similar). The architecture is not optional. Build on ARM64 or a Mac and the compute rejects the binary with Exec format error. There is no cross-arch fallback; the interpreter simply won’t run.

⚠️ Gotcha

Match more than just the CPU. conda-pack ships compiled binaries, so they link against the build box’s system libraries — glibc in particular. Build on a newer OS than the compute (e.g. Ubuntu 24.04 for an older compute image) and you can hit GLIBC_2.xx not found at import time. Build on an OS that’s the same or older than the target, and match the Python patch line, to stay safe.

3.1 Install Miniconda and accept the ToS (once)

curl -fSL -o miniconda.sh https://repo.anaconda.com/miniconda/Miniconda3-latest-Linux-x86_64.sh

bash miniconda.sh -b -p $HOME/miniconda

source $HOME/miniconda/etc/profile.d/conda.sh

conda init bash && exec bash

# Accept the ToS HERE, on the box that can actually reach the endpoint:

conda tos accept --override-channels --channel https://repo.anaconda.com/pkgs/main || true

conda tos accept --override-channels --channel https://repo.anaconda.com/pkgs/r || true

conda install -n base -c conda-forge conda-pack -y

3.2 Create the env (ipykernel is mandatory)

# The env name follows through the whole workflow. Pick your version:

conda create -n aml-py310 -c conda-forge python=3.10 ipykernel pip setuptools wheel -y

conda create -n aml-py314 -c conda-forge python=3.14 ipykernel pip setuptools wheel -y

Include ipykernel. The installer on the compute runs a preflight check for it and aborts with a rebuild message if it’s missing. It comes from conda here, which is why it does not appear in the pip requirements files. Leave it out and you’ll do the whole build, upload, and download before finding out nothing registers.

3.3 Install the curated package set

A note on the requirements file: there’s no magic requirements-py310.txt attached to this post — it stands in for your own curated package list. Bring whatever set your team needs; Section 4 shows how to sanitise it and Section 5 covers the per-version differences. The one-file-per-Python-version layout below is just how I keep the versions straight.

conda activate aml-py310

pip install -r requirements/requirements-py310.txt

pip check # surface residual conflicts BEFORE you pack

# Freeze exact versions so a rebuild next month is byte-for-byte repeatable:

pip freeze > requirements/requirements-py310.lock.txt

On 3.12 / 3.13 / 3.14 this set is built around azure-ai-ml (v2); the deprecated azureml-* v1 family is intentionally absent. Section 5 explains why.

3.4 Pack it

Run this from base (where conda-pack lives), not from inside the env:

conda deactivate

mkdir -p ~/aml-offline && cd ~/aml-offline

conda pack -n aml-py310 -o aml-py310.tar.gz --ignore-missing-files

ls -lh aml-py310.tar.gz # ~150 MB minimal … several GB with the full DS stack

⚠️ Gotcha

--ignore-missing-files is required for these envs. Because the env mixes conda with a large pip layer, pip upgrades a few conda-installed packages (e.g. packaging) and rewrites their dist-info. Without the flag, conda-pack aborts with CondaPackError: Files managed by conda were found to have been deleted/overwritten. The flag packs what’s actually on disk — the consistent pip versions. Add --force if the output file already exists.

3.5 Smoke-test before shipping

mkdir -p /tmp/test && tar -xzf aml-py310.tar.gz -C /tmp/test

/tmp/test/bin/conda-unpack

/tmp/test/bin/python -c "import sys, ipykernel; print(sys.version, ipykernel.__version__)"

file /tmp/test/bin/python # MUST say: ELF 64-bit LSB executable, x86-64

rm -rf /tmp/test

Run the file check every single time. If it doesn’t say x86-64, stop and rebuild now. Shipping it anyway just moves the same failure to the compute, where it takes longer to spot.

4. Cleaning the curated package list – (Depends on the package list , Here we can see some examples)

A raw package list dumped from a curated AML image looks fine at a glance. Installed as-is, it either fails the build outright or, worse, silently breaks the env in ways you won’t notice until a notebook misbehaves three weeks later. I’ve hit six recurring problems in these lists. Here they are, with the fix for each.

Offender	The problem	The fix
conda, conda-content-trust, conda-package-handling, pycosat, boltons, ruamel-yaml-conda	These are components of the base conda install, not project libraries. pycosat needs a C toolchain, and a pip-installed conda can shadow the real one.	Remove all six. Keep ruamel.yaml (the real PyPI build) for YAML.
azure-cli	It’s a command-line app, not a notebook library, and it pins hundreds of exact azure-mgmt-* / azure-storage-* versions. Single biggest source of conflicts.	Remove it. Install the az CLI separately if you need it — don’t let it dictate your SDK versions.
azureml-core + the azureml-* v1 family	Deprecated (EOL June 2026) and version-capped. Binary deps (azureml-dataprep-rslex, dotnetcore2) only ship wheels to CPython 3.11.	Keep for 3.10/3.11 only. On 3.12–3.14, drop it and use azure-ai-ml (v2).
brotlipy	A conda-image artifact superseded by Brotli. On newer Python it often has no wheel and wants a C compiler.	Remove. requests / urllib3 pull in Brotli automatically when needed.
dataclasses==0.6, uuid==1.30	Ancient backports that shadow the standard library on modern Python. These cause real, subtle breakage — not just noise.	Remove on sight.
azure-storage-filedatalake (and duplicates)	That name 404s on PyPI — it’s the import name. azure-identity and databricks-sql-connector also appear twice.	Correct to azure-storage-file-datalake (dashes). Deduplicate the rest.

💡 Tip

Do the resolution out in the open, on the build box, where pip check and error messages are visible and cheap to fix. The whole point of shipping a tarball is that the locked-down compute never runs a solver — so make sure the solver runs cleanly before you pack.

5.The azureml v1 version ceiling

This one deserves its own section because it silently dictates which Python versions you can even offer. The azureml-* v1 SDK — azureml-core and its binary friends azureml-dataprep-rslex, azureml-dataprep-native, dotnetcore2 — is deprecated (EOL June 2026) and only ships wheels for CPython 3.8–3.11 (rslex reaches 3.12, but not 3.13/3.14). On 3.13/3.14 the install simply cannot resolve. So the strategy is version-aware:

Python	Requirements file	*azureml- v1**	Notebook SDK
3.10	requirements-py310.txt	Included	azureml-core
3.11	requirements-py311.txt	Included	azureml-core
3.12	requirements-py312.txt	Dropped	azure-ai-ml
3.13	requirements-py313.txt	Dropped	azure-ai-ml
3.14	requirements-py314.txt	Dropped	azure-ai-ml

The v2 azure-ai-ml SDK is tested on Python 3.8–3.14 and covers workspace, compute, datastore, job, pipeline, and AutoML operations — so notebooks keep working on the newer interpreters without the deprecated stack.

The filenames in this table are just a per-version naming convention for your own package lists — not files you download from anywhere. Keep one list per Python version and the rest of the workflow is identical.

6. Part 2 — Upload the tarball to blob

The tarball is packed and tested. Now drop it in a blob container the compute can reach over its private endpoint. This is where a role-assignment subtlety trips up almost everyone.

6.1 Upload options

Portal (easiest): Storage account → Containers → your container → + Add Directory offline-kernels → Upload the .tar.gz. The Portal uses the account key, so it sidesteps the RBAC issue below entirely.

azcopy with the build VM’s managed identity:

curl -fSL -o azcopy.tar.gz https://aka.ms/downloadazcopy-v10-linux

tar -xzf azcopy.tar.gz

sudo cp azcopy_linux_amd64_*/azcopy /usr/local/bin/ && sudo chmod +x /usr/local/bin/azcopy

azcopy login --identity # add --identity-client-id <uami> for user-assigned

azcopy copy "$HOME/aml-offline/aml-py310.tar.gz" \

"https://<ACCOUNT>.blob.core.windows.net/<CONTAINER>/offline-kernels/aml-py310.tar.gz"

6.2 The 403 that ignores your title

You’re a subscription Owner. Azure is unimpressed ☹. The upload still fails with 403 AuthorizationPermissionMismatch. Owner and Contributor are control-plane roles; they let you manage the storage account. Reading and writing blobs is a data-plane operation, and it needs a data-plane role the control-plane ones don’t include.

I lost most of an afternoon to this during a bank onboarding earlier this year. I was sure the role assignment just hadn’t propagated yet, so I kept re-running the copy and waiting. The real problem was that I’d been granted Contributor on the account and never the data-plane role. About ninety seconds after the right role landed, the upload went through. Check the role before you blame propagation.

Operation	Role you actually need	Plane
Upload the tarball (build side)	Storage Blob Data Contributor	Data-plane
Download the tarball (compute side)	Storage Blob Data Reader	Data-plane
Owner / Contributor	Manage the account — not sufficient for blob I/O	Control-plane

sid=$(az storage account show -n <ACCOUNT> --query id -o tsv)

az role assignment create \

--assignee <object-id-or-MI-principal-id> \

--role "Storage Blob Data Contributor" \

--scope "$sid"

# wait ~1–2 min for RBAC propagation, then retry the copy

⚠️ Gotcha

Whatever account you upload to must be the same one the installer points at, and the compute’s own managed identity needs Storage Blob Data Reader on it. Mismatch the accounts and the download 403s the same way, except now it fails on the compute instead of your build box.

7. Part 3 — Install the kernel on the compute

On the compute’s terminal (AML Studio → the compute → Terminal, where you’re azureuser), point install_offline_kernel.py at the right env and run it. The full script is in the Appendix at the end of this post.

# Constants at the top of install_offline_kernel.py:

STORAGE_ACCOUNT = "<your-account>"

CONTAINER = "data"

BLOB_PREFIX = "offline-kernels"

ENV_NAME = "aml-py310" # aml-py311 / py312 / py313 / py314

DISPLAY_NAME = "Python 3.10 (AML Custom)"

# Only for a user-assigned MI, so IMDS knows which identity to use:

export AZURE_CLIENT_ID="<uami-client-id>"

python3 install_offline_kernel.py

What the script does:

Bootstraps the SDK — installs azure-identity + azure-storage-blob into the user site (~5 MB, one-time). It also supports a local tarball or a SAS URL, so it works with zero Azure auth if you pre-stage the file.
Authenticates via DefaultAzureCredential — picks up the compute’s managed identity from IMDS.
Downloads offline-kernels/aml-py310.tar.gz over the private endpoint.
Extracts into /anaconda/envs/aml-py310 (sudo only if the path isn’t user-writable).
Runs conda-unpack — rewrites the embedded absolute paths so the env works in its new home.
Preflight-checks that the env can import ipykernel, and aborts with an exact rebuild command if not.
Registers the kernel as “Python 3.10 (AML Custom)” for azureuser. Idempotent — it skips the unpack if the env already exists.

Force a clean rebuild after re-uploading a new tarball:

sudo rm -rf /anaconda/envs/aml-py310

rm -rf ~/.local/share/jupyter/kernels/aml-py310

python3 install_offline_kernel.py

Verify: in AML Studio open a notebook → kernel picker → select “Python 3.10 (AML Custom)” and run import sys; print(sys.version). On 3.12+, reach for the v2 SDK: from azure.ai.ml import MLClient.

8. When it doesn’t work

The short list I check first, in order of how often each one is the culprit:

Symptom	Cause	Fix
Exec format error: …/bin/python	Tarball built on the wrong arch/OS (ARM64 or macOS).	Rebuild on Linux x86_64 (Section 3), re-upload, retry.
ERROR: … cannot import ipykernel	Env packed without ipykernel.	Recreate the env WITH ipykernel, repack, re-upload, clean-rebuild.
DefaultAzureCredential failed …	No MI on the compute, or MI lacks the data role.	Attach an MI to the compute (not the workspace) + Storage Blob Data Reader.
HTTP 403 from blob	MI lacks the data-plane role on this account/container.	Add Storage Blob Data Reader on the same account you uploaded to.
HTTP 404 from blob	Wrong account/container/blob path in the script.	Re-check the constants against the actual blob path in the Portal.
Multiple identities from IMDS	User-assigned MI; the SDK can’t guess which one.	export AZURE_CLIENT_ID="<uami-client-id>" first.

⚠️ Gotcha

The single most common face-plant is the workspace-vs-compute identity mix-up. Granting a role to the AML workspace identity does nothing for IMDS calls coming from the compute. The managed identity must be assigned to the compute instance itself. Check that before you check anything else.

9. Why this actually works with no internet

Four details make the whole thing hold together. They’re also exactly what a security reviewer will ask about, so it helps to have the answers ready.

IMDS is not the internet. Managed-identity auth works with zero outbound access because IMDS lives at 169.254.169.254 — a link-local address served by the Azure host, not a public endpoint. The blob download then rides the Storage account’s private endpoint inside the VNet. Nothing touches the public internet.
No Anaconda ToS prompt on the secure side. You accepted the ToS on the build box. The compute only ever sees a plain tarball, so no conda channel is ever contacted and the ToS gate never appears.
Workspace MI ≠ compute MI. RBAC granted to the workspace identity does not authorize IMDS calls from the compute. The identity must live on the compute instance. This is the top cause of a mysterious 403 — worth repeating.
Kernels persist across stop/start. The kernelspec and env live on the persistent user disk (~/.local/share/jupyter/kernels/ and /anaconda/envs/). A stop/start keeps them; only deleting the compute wipes them — at which point you just re-run the installer.

10. conda-pack gotchas on the build box

The build box has its own small collection of traps. Sharing the scars so you can skip them:

Trap	What you see	The fix
python -m conda_pack …	package cannot be directly executed	The module has no __main__. Use the console script conda-pack … or the conda pack subcommand.
conda pack	invalid choice: 'pack'	conda-pack isn’t installed in that conda. conda install -n base -c conda-forge conda-pack -y.
Packing a mixed conda+pip env	Files managed by conda … deleted/overwritten	Add --ignore-missing-files; add --force if the .tar.gz exists.
conda-pack script missing	conda-pack: command not found	python -m pip install conda-pack, then hash -r, then conda-pack -p <env-prefix> -o out.tar.gz.

Wrapping up

You don’t need to punch a hole in the firewall to give a locked-down Azure ML compute a rich, multi-version Python environment. You need to do all the messy dependency resolution outside, where you can see it, freeze the result with conda-pack, ship one tarball over a private endpoint, and register it as a kernel on the other side. Build once, pack once, upload once, install once.

The parts that bite are all predictable once you know they’re there: the arch mismatch, the azureml version ceiling, the data-plane 403, and the workspace-vs-compute identity. Now you know, so they should cost you minutes instead of an afternoon.

Appendix — install_offline_kernel.py

This is the installer referenced in Section 7 — the only script that runs on the air-gapped compute. It fetches the tarball three ways in priority order (a local file, a SAS URL, or blob download via the compute’s managed identity), unpacks it, runs conda-unpack, preflight-checks ipykernel, and registers the kernel. Set the constants at the top, copy it onto the compute, and run python3 install_offline_kernel.py.

📌 Note

Replace the placeholders. The storage account, subscription id, and CLI user below are placeholders — swap in your own values before running. Everything else is ready to use as-is.

#!/usr/bin/env python3

"""

Install an offline conda-packed Python kernel on an AML compute instance.

Gets a conda-pack tarball (in priority order):

1. a local tarball on this box (LOCAL_TARBALL, or ~/aml-offline/<ENV>.tar.gz,

or ./<ENV>.tar.gz) - no Azure auth needed,

2. a SAS URL (BLOB_SAS_URL) - no RBAC data role needed,

3. Azure Blob via DefaultAzureCredential (compute managed identity).

Then unpacks it and registers it as a Jupyter kernel for the current user.

Prerequisites (only for option 3 - blob download via managed identity):

- A managed identity (system- or user-assigned) attached to the compute instance.

- That identity has "Storage Blob Data Reader" on the storage account (DATA-plane

role; control-plane Contributor/Owner is NOT sufficient).

- For user-assigned MI, set env var AZURE_CLIENT_ID to its client id.

"""

from __future__ import annotations

import os

import subprocess

import sys

from pathlib import Path

# ---- Configuration ---------------------------------------------------------

STORAGE_ACCOUNT = "<your-account>"

CONTAINER = "data"

BLOB_PREFIX = "offline-kernels"

ENV_NAME = "aml-py310"

DISPLAY_NAME = "Python 3.10 (AML Custom)"

# If the tarball is already on this machine (e.g. you built it here), the script

# uses it and skips the blob download entirely - no Azure auth / RBAC needed.

# Leave "" to auto-search ~/aml-offline/<ENV_NAME>.tar.gz and the current dir,

# or set an explicit path.

LOCAL_TARBALL = ""

# Optional full SAS URL to the blob, e.g.

# https://<acct>.blob.core.windows.net/<cont>/<prefix>/<env>.tar.gz?sv=...

# If set, it is used instead of DefaultAzureCredential and bypasses the

# "Storage Blob Data Reader" RBAC requirement. Leave "" for managed-identity auth.

BLOB_SAS_URL = ""

# Azure auth for the blob download (only used if no local tarball and no SAS URL).

# CREDENTIAL_MODE = "cli" -> use your signed-in Azure CLI user; first run:

# az login # device-code on a compute

# az account set --subscription $SUBSCRIPTION_ID

# CREDENTIAL_MODE = "managed" -> use the compute's managed identity

# CREDENTIAL_MODE = "default" -> DefaultAzureCredential chain (MI, then CLI, ...)

SUBSCRIPTION_ID = "<your-subscription-id>"

AZURE_USERNAME = "<your-cli-user>" # expected CLI user (informational)

CREDENTIAL_MODE = "cli"

TARGET_DIR = Path(f"/anaconda/envs/{ENV_NAME}")

TARBALL_PATH = Path(f"/tmp/{ENV_NAME}.tar.gz")

# ---------------------------------------------------------------------------

def run(cmd: list[str], **kwargs) -> None:

"""Run a subprocess command, streaming output and raising on failure."""

print(f"+ {' '.join(cmd)}", flush=True)

subprocess.run(cmd, check=True, **kwargs)

def ensure_sdk_packages() -> None:

"""Install azure-identity and azure-storage-blob into the current interpreter."""

try:

import azure.identity # noqa: F401

import azure.storage.blob # noqa: F401

return

except ImportError:

pass

print("Installing azure-identity and azure-storage-blob ...", flush=True)

run([

sys.executable, "-m", "pip", "install", "--quiet", "--user",

"azure-identity", "azure-storage-blob",

])

def find_local_tarball() -> Path | None:

"""Return an existing local tarball to use instead of downloading, or None."""

candidates = []

if LOCAL_TARBALL:

candidates.append(Path(LOCAL_TARBALL).expanduser())

candidates.append(Path.home() / "aml-offline" / f"{ENV_NAME}.tar.gz")

candidates.append(Path.cwd() / f"{ENV_NAME}.tar.gz")

for c in candidates:

if c.is_file():

return c

return None

def get_credential():

"""Return an Azure credential according to CREDENTIAL_MODE."""

if CREDENTIAL_MODE == "cli":

from azure.identity import AzureCliCredential

# Uses the user signed in via `az login` (e.g. <your-cli-user>).

print(f"Using Azure CLI credential (expected user: {AZURE_USERNAME}).", flush=True)

return AzureCliCredential()

if CREDENTIAL_MODE == "managed":

from azure.identity import ManagedIdentityCredential

client_id = os.environ.get("AZURE_CLIENT_ID")

print("Using the compute's managed identity.", flush=True)

return ManagedIdentityCredential(client_id=client_id) if client_id else ManagedIdentityCredential()

from azure.identity import DefaultAzureCredential

print("Using DefaultAzureCredential chain.", flush=True)

return DefaultAzureCredential()

def download_blob(env_name: str, dest: Path) -> None:

"""Download the conda-pack tarball via SAS URL or an Azure credential."""

from azure.storage.blob import BlobClient

dest.parent.mkdir(parents=True, exist_ok=True)

if BLOB_SAS_URL:

print(f"Downloading via SAS URL -> {dest}", flush=True)

client = BlobClient.from_blob_url(BLOB_SAS_URL)

else:

ensure_sdk_packages()

credential = get_credential()

blob_name = f"{BLOB_PREFIX}/{env_name}.tar.gz"

url = f"https://{STORAGE_ACCOUNT}.blob.core.windows.net/{CONTAINER}/{blob_name}"

print(f"Downloading {url} -> {dest}", flush=True)

client = BlobClient.from_blob_url(url, credential=credential)

try:

with open(dest, "wb") as f:

stream = client.download_blob(max_concurrency=4)

stream.readinto(f)

except Exception as exc: # noqa: BLE001

msg = str(exc)

if "AuthorizationPermissionMismatch" in msg or "not authorized" in msg:

sys.exit(

"ERROR: 403 AuthorizationPermissionMismatch downloading the tarball.\n"

f"The credential in use (CREDENTIAL_MODE='{CREDENTIAL_MODE}') lacks the\n"

f"DATA-plane role 'Storage Blob Data Reader' on storage account\n"

f"'{STORAGE_ACCOUNT}'. (azcopy login / control-plane Contributor do NOT grant this.)\n"

"Fix ONE of:\n"

f" 1. Grant that role to the principal (e.g. {AZURE_USERNAME} for CLI mode,\n"

" or the compute's managed identity for managed mode), or\n"

" 2. Set BLOB_SAS_URL to a SAS link (no RBAC needed), or\n"

f" 3. Put the tarball at ~/aml-offline/{env_name}.tar.gz to skip download.\n\n"

f"Original error:\n{msg}"

)

raise

print(f"Downloaded {dest.stat().st_size:,} bytes", flush=True)

def ensure_target_dir(target: Path) -> None:

"""Create target dir, using sudo if necessary, and chown to current user."""

if target.exists():

return

user = os.environ.get("USER", "azureuser")

try:

target.mkdir(parents=True, exist_ok=True)

except PermissionError:

print(f"Need sudo to create {target}", flush=True)

run(["sudo", "mkdir", "-p", str(target)])

run(["sudo", "chown", "-R", f"{user}:{user}", str(target)])

def install_kernel() -> None:

if (TARGET_DIR / "bin").is_dir():

print(f"Env {ENV_NAME} already exists at {TARGET_DIR}, skipping unpack.", flush=True)

else:

local = find_local_tarball()

if local is not None:

print(f"Using local tarball {local}; skipping blob download.", flush=True)

src = local

downloaded = False

else:

download_blob(ENV_NAME, TARBALL_PATH)

src = TARBALL_PATH

downloaded = True

ensure_target_dir(TARGET_DIR)

print(f"Extracting {src} -> {TARGET_DIR}", flush=True)

run(["tar", "-xzf", str(src), "-C", str(TARGET_DIR)])

print("Running conda-unpack ...", flush=True)

run([str(TARGET_DIR / "bin" / "conda-unpack")])

# Only remove the temp download, never a user-provided local tarball.

if downloaded:

try:

TARBALL_PATH.unlink()

except FileNotFoundError:

pass

print(f"Registering Jupyter kernel '{ENV_NAME}' ...", flush=True)

env_python = str(TARGET_DIR / "bin" / "python")

# Sanity-check: env must actually contain ipykernel, otherwise the tarball

# was built without it and needs to be rebuilt on the Linux build box.

check = subprocess.run(

[env_python, "-c", "import ipykernel, sys; print(ipykernel.__version__)"],

capture_output=True, text=True,

)

if check.returncode != 0:

sys.exit(

f"ERROR: {env_python} cannot import ipykernel.\n"

f"The conda-pack tarball was built without ipykernel. Rebuild on the Linux box with:\n"

f" conda create -n {ENV_NAME} -c conda-forge python=<major.minor> ipykernel pip setuptools wheel -y\n"

f" conda-pack -n {ENV_NAME} -o {ENV_NAME}.tar.gz --ignore-missing-files\n"

f"then re-upload and clean up: sudo rm -rf {TARGET_DIR}\n\n"

f"stderr from check:\n{check.stderr}"

)

print(f"ipykernel {check.stdout.strip()} found in env.", flush=True)

run([

env_python, "-m", "ipykernel", "install",

"--user",

"--name", ENV_NAME,

"--display-name", DISPLAY_NAME,

])

def main() -> int:

print(f"Starting offline {ENV_NAME} kernel setup on AML compute instance ...", flush=True)

install_kernel()

kernels_dir = Path.home() / ".local" / "share" / "jupyter" / "kernels"

if kernels_dir.is_dir():

print(f"\nInstalled kernels in {kernels_dir}:", flush=True)

for k in sorted(kernels_dir.iterdir()):

print(f" - {k.name}")

print("\nOffline kernel setup completed.", flush=True)

return 0

if __name__ == "__main__":

sys.exit(main())

References

From Downtime Risk to Enterprise Resilience: SAP NetWeaver Java HA Journey with SUSE

AnuradhaKarnam — Tue, 07 Jul 2026 17:35:51 GMT

Introduction:

For years, the IT team at a global manufacturing company had relied on SAP NetWeaver 7.50 Java to support critical business processes. The environment handled everything from employee self-service applications to business integrations across multiple regions. While the infrastructure was stable, one concern remained constant: SAP Central Services instance represented a potential single point of failure.

Every maintenance window required careful planning. Any unexpected server outage could disrupt business operations, impact users worldwide, and create pressure on support teams to restore services as quickly as possible.

Challenge:

The SAP landscape was growing, and business stakeholders expected continuous availability. Traditional backup and recovery procedures could restore services, but you could not eliminate the downtime associated with infrastructure failures.

The architecture team identified SAP Java Central Services as a critical component that needed enhanced protection.

The objective was simple:

Eliminate single points of failure.
Automate failover during infrastructure issues.
Minimize business disruption.
Improve operational confidence.

Solution:

You adopted a SUSE High Availability solution built around clustered architecture. Multiple nodes were configured to work together, continuously monitoring the health of SAP services and infrastructure resources.

Using SUSE HA, the environment could automatically detect failures and transfer workloads to a healthy node whenever required. Instead of relying on manual intervention, recovery became a coordinated and automated process.

The architecture provided:

Continuous monitoring of critical SAP components.
Automated failover capabilities.
Resource management through cluster policies.
Improved service continuity during unplanned outages.

Implementation:

Deployment required close collaboration between SAP administrators, Linux engineers, and infrastructure architects. You carefully validated cluster configurations, tested failover scenarios, and documented operational procedures.

Several disaster recovery simulations were conducted:

Server shutdown tests.
Network interruption scenarios.
Service failure simulations.
Planned maintenance activities.

Each test demonstrated how the cluster could maintain application availability while reducing recovery times.

Results:

As the solution moved into production, confidence across the organization increased significantly. Business users experienced uninterrupted access to critical SAP services, while operations teams gained greater visibility into system health and cluster status.

The organization realized several key benefits:

Higher platform availability.
Faster recovery from failures.
Reduced operational risk.
Improved maintenance flexibility.
Enhanced business continuity.

Lessons Learned:

The project reinforced an important principle for enterprise SAP environments:

High availability is not simply about recovering from failure, it's about designing systems that continue operating when failures occur.
By combining SAP NetWeaver 7.50 Java Central Services with SUSE High Availability technology, the company transformed a potential business risk into a resilient and reliable platform capable of supporting mission-critical operations around the clock.

Key Benefits of SUSE HA for SAP NetWeaver Java Central Services

Elimination of Single Points of Failure
SAP NetWeaver Java Central Services (SCS) is a critical component of the SAP Java stack. SUSE High Availability (HA) protects the SCS instance by clustering resources across multiple nodes, reducing the risk of service interruption caused by server failures.
Automated Failover and Faster Recovery
When a node or service fails, SUSE Pacemaker automatically detects the issue and transfers SAP services to a healthy node. This minimizes manual intervention and significantly decreases recovery time.
Increased Business Continuity
By maintaining service availability during infrastructure outages, SUSE HA helps ensure that business-critical SAP applications remain accessible to users, reducing operational disruptions.
Reduced Downtime During Planned Maintenance
Organizations can perform maintenance activities on cluster nodes with minimal impact to SAP services. Workloads can be shifted between nodes, helping ensure uninterrupted operations.
Continuous Monitoring and Health Checks
SUSE HA continuously monitors cluster resources, SAP processes, virtual IP addresses, and storage components. Early detection of issues helps improve overall system reliability.
Improved Operational Efficiency
Automated resource management reduces the need for manual failover procedures and allows administrators to focus on proactive platform management rather than reactive troubleshooting.
Enhanced System Resilience
The cluster architecture is designed to withstand hardware, operating system, network, and service failures, creating a more robust SAP landscape.
Better Disaster Recovery Readiness
Regular failover testing and cluster validation capabilities help organizations verify their recovery procedures and improve preparedness for unexpected outages.
Enterprise-Grade Reliability for Mission-Critical Workloads
For organizations running critical SAP business processes, SUSE HA provides the foundation for achieving high availability and meeting stringent uptime requirements.

Conclusion:

In modern SAP landscapes, resilience is a business requirement rather than a technical luxury. SUSE High Availability provides organizations with a robust foundation for protecting SAP NetWeaver Java workloads, reducing downtime, and ensuring that critical business services remain available when they are needed most. SUSE High Availability for SAP NetWeaver 7.50 Java Central Services enables organizations to build a resilient SAP platform by eliminating single points of failure, automating failover, and minimizing downtime. Through continuous monitoring, intelligent resource management, and cluster-based protection, businesses can improve service continuity, simplify operations, and ensure that mission-critical SAP workloads remain available when they are needed most.

Reference links:

SUSE HA for SAP NetWeaver 7.50 Java central services | SUSE | Support Center

For the main cluster configuration and the automatic mounting of shared file systems during startup using entries defined in /etc/fstab, see the "High Availability Cluster with Simple Mount" section at the link below:

https://documentation.suse.com/sbp/sap-15/html/SAP-S4HA10-setupguide-simplemount-sle15/index.html

Refer to the resource groups outlined in the TID, together with the necessary SAP profile file modifications and location constraints documented in the "SAP NetWeaver Enqueue Replication 1 High Availability Cluster – SAP NetWeaver 7.40 and 7.50" guide linked below. These configurations are applicable since the Java SCS instance supports Enqueue Replication 1 (ENSA1) only:

https://documentation.suse.com/sbp/sap-15/html/SAP-nw740-sle15-setupguide/index.html

Azure OpenAI Architecture: The Decisions That Actually Matter (Part 3)

luciacasucci — Wed, 17 Jun 2026 16:39:08 GMT

Introduction

Part 1 of this series tackled the architectural decisions that shape any Azure OpenAI / Microsoft Foundry Models workload — capacity model, deployment scope, governance layer, grounding strategy, and quota engineering. Part 2 turned those decisions into a Well-Architected Framework discipline. Part 3 looks at the part that makes GenAI architecture genuinely different from a traditional service: the platform itself never stops moving.

Models are released, promoted to GA, moved to Legacy, deprecated, and eventually retired. New regions come online; certain features (such as Priority Processing) light up only on specific model versions and deployment scopes. Fine-tuned models inherit the lifecycle of their base. Performance characteristics shift between releases. Reliability in this world is not just uptime — it is the ability to absorb continuous change without disrupting production.

That discipline is GenAIOps: the people, processes, and tooling that turn model upgrades from emergency events into routine operations. Part 2 already covers the core lifecycle mechanics and upgrade policy trade-offs through a Well-Architected lens. Part 3 stays focused on the operational and architectural practices that make change safe: evaluation of pipelines, observability, routing patterns, prompt governance, and abstraction. Where details are time-sensitive — stage thresholds, SLA windows, regional rollout delays, capacity tier eligibility — they are flagged with "At the time of writing". Always confirm current behavior against Microsoft Learn before committing to a design.

Who is this series for?

Cloud and Solution Architects
Platform and product owners
Senior developers responsible for operating Azure OpenAI workloads in production

What you’ll learn in Part 3:

How to build an evaluation pipeline that promotes model upgrades the way CI/CD promotes code.
How to instrument full-stack observability so regressions surface early (latency, errors, token trends, quality drift).
How the Model Router pattern, canary releases, and tier-aware fallbacks turn model change into a configuration concern.
How to govern prompts as production artifacts with versioning, feature-flagged rollouts, and regression testing.
How to manage lifecycle-dependent assets (fine-tuned models) and regional rollout realities without firefighting — plus a GenAIOps Decision Matrix you can reuse as a checklist.

We’ve also included a summary decision matrix at the end of this post for quick reference.

1. Model lifecycle (recap)

Azure OpenAI/Microsoft Foundry models are living dependencies: new versions are released, promoted from Preview to GA, then eventually move through deprecation toward retirement. To avoid surprises, treat every deployed model version as having an expiration date and design so you can swap versions without rewriting application code. In general, use the Standard deployment auto-upgrade mode that preserves stability but guarantees continuity at retirement, and plan to deliberate blue/green migrations for dedicated (provisioned) capacity where auto-upgrade is not available. For the deeper mechanics (upgrade modes, retirement behavior, and migration playbooks), refer to Part 2’s Reliability section; the rest of this article focuses on the GenAIOps practices that make those upgrades routine.

Figure 1 — Models lifecycle

2. GenAIOps: Evaluating Before Promoting

Upgrading a model should not be a manual, subjective exercise. Azure AI Foundry provides evaluation capabilities that, combined with a regression prompt suite, turn model upgrades into measurable, repeatable decisions:

Side-by-side prompt comparisons across model versions.
Automated quality scoring (relevance, coherence, groundedness, safety, and fluency).
Structured-output validation (JSON conformance, schema validation).
Batch testing across comprehensive prompt libraries representative of real production traffic.
Custom evaluation metrics tailored to your domain.

Architectural best practice:

Maintain a curated regression prompt suite that mirrors real production traffic — including the long tail.
Run evaluation pipelines against candidate models before any production cut-over.
Integrate evaluation into CI/CD using Azure DevOps, GitHub Actions, or similar automation.
Define quality gates that must pass before promotion (e.g., groundedness ≥ a target threshold, p95 latency under a target budget). Pick numbers that fit your workload, not the article.

Model promotion should require passing the evaluation gates the same way application code requires passing unit tests. Without automated evaluation, model upgrades become high-risk, low-visibility events that teams avoid until forced by retirement deadlines — the exact pattern that keeps lifecycle work in the "emergency" column instead of the "scheduled" column.

Example evaluation workflow:

Trigger — a new model version reaches GA, or your migration playbook hits the R-90 step.
Deploy — the candidate model goes to a staging deployment.
Regress — the prompt suite (typically several hundred to several thousand prompts) is run against the candidate.
Compare — the candidate's outputs are scored against the current production model.
Inspect — humans review flagged differences; metrics, latency distributions, and cost-per-request go on the dashboard.
Gate — an approval step (manual or automated) decides whether the candidate proceeds to blue/green production deployment.

Figure 2 — Evaluation Pipeline

3. Observability: Full-Stack or It Didn't Happen

GenAIOps is more than one-time evaluation. Once a candidate's model has been promoted, you need continuous, end-to-end observability across the request path — not just at the model boundary. Without it, you are operating blind during model transitions.

At a minimum, instrument:

Prompt processing time (gateway through model invocation).
Model inference latency, expressed as p50, p95, and p99 — averages hide the experience of the slowest 5% of users.
Token consumption (prompt tokens, completion tokens, total) trended over time.
Error rates by class (429 throttling, 503 service unavailable, 400 validation errors, content-filter rejections).
Model version distribution — which versions are actually serving traffic right now.
User-satisfaction signals (thumbs-up/down, explicit feedback, session abandonment).

Many performance regressions only surface at scale. A model version that performs well in evaluation against a few hundred prompts may behave differently under production traffic patterns. Plan for that.

A practical metrics architecture on Azure tends to combine:

Application Insights for end-to-end request tracing across the application and gateway.
Azure Monitor for infrastructure, quota, and PTU utilization of metrics.
Custom telemetry for prompt-level success/failure tracking and quality scoring.
Log Analytics for forensic analysis when a regression is suspected.

Drift in model behavior rarely shows up as a single broken request — it surfaces as a slow shift in tail latency, fallback rate, or user-satisfaction signal. Monitoring that only looks at average will miss it.

4. The Model Router Pattern

As GenAI systems mature, a static single-model architecture becomes both limiting and expensive. A Model Router introduces dynamic, intelligent model selection in front of one or more model deployments.

Typical responsibilities of a router:

Send simple queries to a smaller, faster model and complex reasoning to a larger one.
Run canary releases of new model versions with percentage-based rollouts.
A/B test model variants to measure quality, latency, and cost differences.
Route to the right capacity tier — including falling back from Provisioned to Standard during migrations or capacity constraints.
Where the workload also needs lower-variance latency on the Standard side, route latency-critical traffic through Priority Processing on a Global Standard or Data Zone Standard (US) deployment, on a model version that supports it. (At the time of writing, Priority Processing is enabled by setting the service_tier attribute on the request and requires a model version released on or after 2025-12-01 — verify both eligibility constraints on Microsoft Learn before depending on it.

Decision logic can be driven by any combination of:

Query complexity (simple heuristics or a lightweight classifier).
User tier (e.g., free vs premium).
Response-time requirements (interactive vs background).
Cost constraints — pick the cheapest model that meets the quality bar.
Regional model availability and capacity.

Implementation options:

Azure API Management — built-in routing policies, weighted backends, retry policies.
Azure Front Door — global routing with health probes.
Custom routing service — maximum flexibility, more operational overhead.
Semantic Kernel or LangChain — framework-level routing logic embedded in the application.

Beyond cost and performance, the Model Router pattern decouples the application layer from any single model version. That decoupling is what makes lifecycle management tractable: when a model moves to Legacy, you change a router rule, not application code.

Figure 3 — Model Router Architecture vs Blue/Green Deployment

5. Prompt Lifecycle Governance

Prompts are not strings embedded in code. They are production artifacts that influence quality, cost, and safety, and they evolve almost as often as the models behind them. Treat them as first-class assets.

Prompt templates

Separate stable system instructions from dynamic content (user input and retrieved context). This lets you version, test, and audit each layer independently.

Version control

Store prompts in Git — full history, code review, branching, and tagging. Treat prompt changes the way you treat code changes: pull request, review, and test before merging.

Feature-flagged rollouts

Roll out prompt changes gradually using feature flags. Monitor the impact on a subset of users before exposing the change broadly. The same observability stack that watches model upgrades should watch prompt rollouts.

Regression testing

Maintain a regression suite of expected prompt behaviors and run it whenever prompts or models change. The suite reuses the same evaluation pipeline you built in Section 2.

Prompt-level metrics

Success rate — did the prompt achieve its intended outcome?
Fallback rate — how often did users rephrase or abandon?
Satisfaction score — explicit user feedback.
Token efficiency — average tokens per successful completion (a leading indicator of cost regression).

PII and privacy safeguards

Customer prompts and completions are not used to train base models. That means logging is safe for debugging — but defense in depth still applies:

Redact PII (names, emails, phone numbers, addresses) before logs are written.
Apply RBAC on Log Analytics workspaces so only the right roles can access raw prompt data.
Govern data retention with automated purging after a defined window.
Keep audit trails of who accessed which logs and when.

Prompt quality is not a one-time effort. It is an ongoing operational discipline that needs tooling, processing, and measurement, in the same way application code does.

Figure 4 — Prompt Lifecycle Governance

6. Fine-Tuned Models: The Hidden Retirement Risk

Fine-tuned models inherit the lifecycle of their base model. That creates a cascading retirement risk that many teams overlook.

During base-model deprecation:

New fine-tuning jobs against that base are blocked — you can no longer create new fine-tuned versions.
Existing fine-tuned deployments continue serving inference, with no immediate impact.

When the base model is retired:

Fine-tuned deployments stop responding (HTTP 404), exactly like any other deployment pinned to a retired version.

The migration imperative is straightforward: retrain fine-tuned models on the successor base model well before the retirement date, ideally during the predecessor's Legacy or Deprecated stage.

Architectural considerations:

Track base-model dependencies explicitly in your asset inventory — the same place you track library and runtime versions.
Schedule retraining workflows aligned with base-model lifecycle dates, not with team availability.
Validate fine-tuned model quality on the new base; behavior can shift between base versions.
Keep training datasets in version-controlled storage, so retraining is reproducible.
Re-evaluate whether fine-tuning is still necessary; newer base models, combined with better prompting (few-shot, chain-of-thought, structured outputs), sometimes remove the need entirely.

Common mistake: investing heavily in fine-tuning without budgeting for the recurring retraining cost and lifecycle overhead. Improved prompting on a newer base model is often the cheaper path.

7. Regional Rollouts and Multi-Region Strategy

Successor models are not always available in every Azure region simultaneously. Microsoft typically releases a new version in a subset of regions first, with broader rollout following over weeks or months. At the time of writing, the regional rollout schedule is published per model on Microsoft Learn — confirm before assuming a particular region will receive a release on a particular day.

Maintain staging deployments in early-release regions

Even if production runs elsewhere, maintain a staging deployment in regions that tend to receive new models earliest. That gives you visibility into the successor's behavior before it auto-upgrades into your primary region.

Pre-test successor models before primary auto-upgrades

If your production deployment uses "Once the current version expires", the upgrade will happen automatically. Pre-testing in an early-release region lets you catch behavioral changes before they hit live traffic.

Multi-region routing for lifecycle flexibility

Azure Front Door or Azure API Management with multi-region back-ends lets you route based on model availability, capacity headroom (one region may have quota while another is exhausted), and latency. Combined with the Model Router pattern from Section 6, this turns regional staggering from a constraint into an option.

Account for capacity-tier eligibility in your routing

Some capacity tiers are scoped to specific deployment scopes — Priority Processing, for example, is offered on Global Standard and Data Zone Standard (US) deployments at the time of writing. Bake those eligibility constraints into routing rules, so a fallback path does not silently land in an ineligible deployment.

Multi-region strategy is no longer just a disaster-recovery concern. It is also lifecycle resilience — the ability to test, stage, and absorb model changes without coupling your platform to a single region release schedule.

8. Future-Proofing Through Abstraction

Future-proofing is architectural, not procedural. The goal is to design systems that adapt to change without requiring code rewrites every time a model is promoted, deprecated, or retired.

Abstract model calls behind a service layer

Avoid calling Azure OpenAI APIs directly from the application code. Introduce an internal Model Service that owns model selection, retry and fallback, prompt-template lookup, and response validation. The application asks for an outcome ("summarize this", "classify that"); the Model Service decides which model and which prompt to use.

Externalize model names and configuration

Store model identifiers, versions, and parameters in configuration or feature flags — never as hard-coded strings. Changing models then becomes a configuration change, not a deployment.

Centralize prompt logic

Maintain prompts in a registry or template repository, not scattered across codebases. This enables centralized versioning, A/B testing without code changes, and prompt optimization that is decoupled from application releases.

Avoid scattering model identifiers across the codebase

Use constants, enums, or configuration references rather than literal model strings repeated across many files. The number of files that have to change at upgrade time is a leading indicator of how painful the upgrade will be.

Benefits of abstraction:

Seamless model replacement — swap models without touching application logic.
Multi-model strategies — the Model Router pattern becomes trivial to add.
Provider flexibility — integrating additional or alternative providers becomes a service-layer change, not an application to rewrite.
Faster adoption of new capabilities — reasoning controls, function calling, structured outputs land in one place.

Common mistake: Prototyping with direct API calls for speed and never refactoring. The technical debt accumulates until a model upgrade requires an emergency engineering sprint.

Figure 5 — Abstraction Layer for Future-Proofing.

Final Perspective

The most important shift this article asks for is a change in operational mindset:

Model upgrades are not emergencies. They are scheduled events.
Retirement deadlines are not surprising. They are published timelines, often with months of notice.
Architecture fails when teams treat models as static dependencies. They succeed when they treat models as evolving infrastructure.

In practice, GenAIOps means:

Automated evaluation that runs continuously, not just during migrations.
Controlled rollouts using blue/green or canary patterns.
Observability-driven decisions based on metrics, not intuition.
Lifecycle-aware planning, with retirement dates tracked alongside library and runtime upgrades.
Modular design that decouples applications from specific model versions.

Across the three parts of this series we have covered the architectural decisions that frame an Azure OpenAI / Microsoft Foundry Models workload (Part 1), the Well-Architected Framework discipline that keeps it sustainable (Part 2), and the GenAIOps practices that let it evolve without firefighting (Part 3). The organizations that succeed long-term are the ones that plan for model evolution from day one, invest in evaluation and observability tooling, decouple application logic from model specifics, and treat prompts and configurations as versioned artifacts.

Generative AI architecture is not about deploying a model endpoint. It is about building a platform that absorbs change gracefully as the AI landscape shifts. The retirement of a model should be a routine operational event, not a crisis. If your architecture makes model upgrades feel risky or expensive, refactor before the next retirement deadline forces your hand.

Lifecycle & GenAIOps Decision Matrix

Use this as a checklist when reviewing or signing off on the GenAIOps posture of an Azure OpenAI / Microsoft Foundry Models platform. One row per decision; one rule of thumb per row.

Area	Decision	Rule of thumb	Watch out for
Lifecycle	Version expiry tracking	Treat model versions as expiring dependencies: inventory every deployed model/version, track deprecation/retirement dates, and design so swapping versions is a configuration change (details on upgrade modes in Part 2).	Pinning versions without an owner; discovering retirement dates after an outage or emergency migration window.
Evaluation	Promotion gates	Pass the regression suite + meet domain-specific quality and latency thresholds before promoting any model.	Subjective "feels better" sign-off; gates that exist on paper but never block a release.
Evaluation	Pipeline integration	Evaluation runs in CI/CD on every candidate; the same suite watches prompt changes.	Manual evaluation runs that only happen under retirement pressure.
Observability	Latency and error metrics	Track p50/p95/p99 latency, 429/503/4xx rates, token trend, and model-version distribution. Alert on tail latency and sustained throttling.	Average-only dashboards; missed Service Health notifications for model retirements.
Observability	Quality drift	Trend per-prompt success rate, fallback rate, and user-satisfaction signals; surface drift before users complain.	Treating quality as a one-time evaluation event.
Architecture	Model Router	Centralize model selection, canary, and fallback (including Priority Processing on eligible deployments) behind a router service.	Application code that calls a specific model deployment by name; routing logic scattered across services.
Architecture	Abstraction layer	Application code asks for an outcome; the Model Service decides which model and prompt; configuration drives model selection.	Hard-coded model identifiers across many files; bypass paths that skip the service layer.
Prompts	Prompt governance	Prompts in Git, behind feature flags, with regression tests, prompt-level metrics, and PII redaction in logs.	Prompts copy-pasted across services; PII in logs; no rollback path for a regressed prompt.
Fine-tune	Fine-tuned model lifecycle	Track fine-tuned models against base-model dates; schedule retraining during the predecessor's Legacy/Deprecated window.	Treating fine-tuned models as permanent infrastructure; lost or unversioned training datasets.
Regional	Multi-region for lifecycle resilience	Maintain staging in early-release regions; route across regions to absorb staggered rollouts and capacity gaps.	Single-region production with no early-release staging; routing rules that ignore tier-eligibility constraints.

Disclaimer

I am a Microsoft employee. The views and opinions expressed in this article are my own and do not necessarily reflect those of Microsoft. This content is informational and educational; it is not an official Microsoft statement, recommendation, or commitment. Service tiers, model availability, lifecycle stages, deprecation timelines, regional rollouts, pricing, and SLAs evolve — always validate against the latest Microsoft Learn documentation before making architectural or migration decisions.

References

Power Platform tenant inventory — a community showcase of the API-first management surface

PravinT — Mon, 08 Jun 2026 13:54:14 GMT

Source: github.com/SweetsNSavories/VerseOps · MIT.

Figure 1 — VerseOps loaded against a live tenant. Per-row capacity (DB / File / Log / FinOps DB / FinOps File GB) is computed from the BAP $expand=properties.capacity call; per-env asset counts are joined client-side from the Inventory API result set. Tenant identifiers redacted; everything else is real.

Figure 2 — One environment row expanded. The row-details template fans out the inventory: Solutions / Apps / Flows / Agents (joined from the Inventory API and per-env Dataverse Web API calls), Power Pages sites (mspp_website table on the env's Dataverse), and the env's systemusers (with their assigned licenses joined from Microsoft Graph). All asset counts (9 / 3 / 53 / 241) are real.

Figure 3 — Total Assets drawer (click the Total Assets KPI tile). The whole panel is fed by a single tenant-wide Inventory API query; the per-type counts are computed client-side from assetType. The "most recent" name surfaces the freshest asset of each kind so an admin can sanity-check that the tenant feed is current.

Figure 4 — Licenses Consumed drawer (click the Licensed Users KPI tile). The list is the union of every assigned servicePlan from /users?$select=assignedLicenses rolled up to the SKU level using the tenant's subscribedSkus catalog from Microsoft Graph. SKU codes are public; the only tenant-specific data is the per-SKU seat count on the right.

Why this exists

The recurring questions at the start of every governance cycle are well known:

"How many environments do we actually have? Who owns the apps in them? How much Dataverse capacity is sitting in places no one remembers creating? Which makers left the company three months ago and still own production flows?"

The official answers — Power Platform admin center (PPAC), the Power Platform inventory page, and the Usage page — already exist and are the right starting point for daily work; they cover the common cases comprehensively. There are still moments, however, when an administrator needs:

A single offline snapshot they can search, sort, filter, and ship to a stakeholder without exposing the live admin center.
A diff between this morning and last Friday — what changed?
Joined views that the portal doesn't ship out of the box: per-env capacity × per-env asset count × per-env user count, all in one sortable grid.
The raw JSON behind every row, one click away, when something doesn't match what the portal shows.
A starting point — code they can fork, instrument, and turn into the governance tool they actually wanted.

VerseOps targets that long tail. The UI surface is roughly five files; every outbound call is enumerated in docs/network-endpoints.md. The codebase is deliberately small, read-only by design, and positioned as a foundation that adopting teams are expected to fork, instrument, and extend.

How it complements the official "Inventory" and "Usage" pages

Microsoft's Power Platform inventory gives administrators a unified view of agents, apps, and flows tenant-wide, refreshed within ~15 minutes. The Usage page tracks engagement and adoption. Both ship in the admin center today and should be every admin's first stop.

VerseOps is positioned as a complement, not a replacement:

Need	PPAC Inventory / Usage	VerseOps
Daily inventory browsing in a portal	✅ Recommended	n/a
Filter / sort / search on any column	✅	✅
Resource-detail drill-in (owner, env, dates)	✅	✅
Export to Excel	✅	✅ (CSV / cache copy)
Capacity (DB / File / Log / FinOps GB) joined per env on the same row as asset count	Partial	✅
One-click "show me the raw Dataverse / PPAC JSON" inspector	❌	✅
Local SQLite cache for offline browsing on a plane / in an air-gapped review	❌	✅
Diff between today's snapshot and yesterday's	❌	✅ (cache-based, on roadmap)
Source you can fork	n/a	✅ MIT, single solution
Telemetry sent to Microsoft / vendor	per Microsoft's data policy	None — zero outbound calls beyond Microsoft's own APIs

If you only ever need 1–4 above, stay in the admin center; it's faster and always up to date. VerseOps shows up when you need 5–11.

Architecture in one diagram

Key architectural properties:

Single process, no server-side footprint. Every call runs in the signed-in user's security context. There is no daemon, no sync job, no message bus. The operating system schedules the network calls; the user triggers a refresh.
Two distinct cloud planes. Management-plane calls (api.powerplatform.com, api.bap.microsoft.com, graph.microsoft.com) are kept separate from data-plane calls ({org}.crm.dynamics.com per environment), with audience switching handled centrally by the auth layer.
The local SQLite database is the only state. Removing %LOCALAPPDATA%\VerseOps\ returns the application to a blank slate. No other persistence exists.

What's actually feasible with the Power Platform API today

Microsoft has been very public about its shift from a UX-first to an API-first development model for Power Platform programmability: new capabilities ship in the API first, then propagate to SDKs, CLI, PowerShell cmdlets, and connectors. The Programmability and extensibility overview lays out the full toolchain — REST API, .NET SDK (Microsoft.PowerPlatform.Management), Python SDK, Power Platform CLI, PowerShell cmdlets, and the Power Platform for Admins V2 connector.

VerseOps is a deliberately small showcase of what the .NET SDK + Inventory API combination unlocks once you put a UI on it:

Capability	API used	SDK / endpoint
List every environment in the tenant with name / region / SKU / version / security group / default-flag	Power Platform API (PPAC)	Microsoft.PowerPlatform.Management SDK
Per-tenant capacity (DB / File / Log / FinOps DB / FinOps File GB)	Power Platform API (PPAC)	SDK Licensing.Tenant.GetCurrentCapacityAllocations()
Per-environment capacity in one tenant-wide call	BAP capacity (legacy GA)	GET /providers/Microsoft.BusinessAppPlatform/scopes/admin/environments?api-version=2020-10-01&$expand=properties.capacity
Every canvas app, model-driven app, code app, cloud flow, agent flow, and Copilot Studio agent in the tenant in one POST	Inventory API (preview)	POST https://api.powerplatform.com/resourcequery/resources/query?api-version=2024-10-01 (KQL-style query against PowerPlatformResources)
DLP policies + connector classification (Business / Non-Business / Blocked)	BAP Governance v2	GET /providers/PowerPlatform.Governance/v2/policies?api-version=2018-01-01
Per-env solutions / Power Pages sites / system users / roles / app + flow status	Dataverse Web API v9.2	GET {org}/api/data/v9.2/solutions, appmodules, workflows, canvasapps, systemusers, mspp_websites
User license SKU resolution + security-group display names	Microsoft Graph	GET /v1.0/subscribedSkus, /users, /groups, /directoryObjects/getByIds

The headline shape of this: one tenant-wide POST replaces what used to be N×6 per-environment GETs. For a tenant with 700 environments, that's the difference between ~4,000 round-trips per refresh and ~10. The same Microsoft.PowerPlatform.Management SDK that powers the new admin-center surfaces is the same one your tooling uses — there's no longer a "fast official one and a slow community one".

A note on the BAP API deprecation path

Several BAP routes the community has relied on for years are now in a clear consolidation track rather than a deprecation one — but the destination is the same. From the official Versioning and support page:

"The 2020-10-01 Generally available version of Power Platform API is specific to environment management and is also commonly referred to as Business Application Platform (BAP) API. The functionality of this set of endpoints are made available in the newer versions of Power Platform API along with many additional features after version 2022-03-01-preview."

In practice, what this means for tools like VerseOps:

BAP route VerseOps uses today	Status (May 2026)	Modern equivalent on api.powerplatform.com
/scopes/admin/environments?$expand=properties.capacity	GA (api-version=2020-10-01); functionally superseded but still recommended for tenant-wide capacity	Will move to a Licensing namespace endpoint as parity completes; track Programmability what's new
PowerPlatform.Governance/v2/policies (DLP)	Stable	Watch the new Connectivity / Governance namespace endpoints (e.g. List Connectors, shipped July 2025)
Microsoft.BusinessAppPlatform provider routes	All being mirrored under api.powerplatform.com namespaces (Licensing, EnvironmentManagement, AppManagement, Authorization, Governance, Connectivity)	Use the SDK — Microsoft maintains the mapping for you

Microsoft's official guidance is unambiguous: use the Power Platform API surface (api.powerplatform.com) and one of the official SDKs (.NET, Python, CLI, PowerShell, Power Platform for Admins V2 connector) for any new automation. BAP routes won't disappear without a deprecation cycle, but new features ship to api.powerplatform.com first and may never come back to BAP.

VerseOps reflects this exactly: every new feature added since April 2026 went to api.powerplatform.com, the BAP capacity client is isolated to a single ~150-line file (BapCapacityClient.cs) so it can be swapped out the moment the per-env capacity surface lands on the new API, and the token-acquisition layer (AuthService.cs) supports both audiences side by side until that day comes.

Who this helps

The MIT license permits unrestricted internal adaptation; adopting teams are encouraged to fork, re-brand, and re-sign the binary with their own enterprise code-signing certificate as part of internal distribution. Typical adopters include:

Power Platform administrators running quarterly governance reviews who need a single defensible snapshot of current tenant state.
Center-of-Excellence (CoE) leads who previously relied on the CoE Starter Kit and are moving to the in-product Inventory + Usage pages, but still require a code-level surface to extend.
FinOps and capacity owners identifying the ~5% of environments that consume 80% of Dataverse storage, with FinOps DB / FinOps File / Log GB visible on the same row as the environment name.
Mission-critical and regulated workloads (financial services, healthcare, public sector) where a desktop tool that authenticates as the human administrator, emits zero telemetry, and stores all state locally is materially easier to risk-accept than a SaaS dashboard.
Security and penetration-test teams who require a reproducible, auditable, signed Windows binary and a clear inventory of what it touches. The SBOM, SECURITY.md, SIGNING.md, and CodeQL workflow are committed to the repository.
Engineering teams learning the Power Platform API who want a non-trivial, well-commented .NET sample that exercises every major namespace.

Where this could go next

The same API surface that powers VerseOps today can support a substantially richer set of experiences. Candidate directions follow; community input on prioritization is welcome via the issue tracker.

1. An agentic governance assistant

Wrap the local SQLite cache + the same auth pipeline behind a Microsoft 365 Copilot agent (or a Foundry agent), and let an admin ask things like:

"Which environments grew the most this week and who owns the new flows?"
"List every canvas app with a deprecated connector that's still 'On' in a production env."
"Show me orphaned resources owned by users disabled in Entra in the last 30 days."

The Power Platform API + Inventory API already returns everything you need to answer these in seconds. The agent surface is just a new face for the same data — and because the cache is local, the agent can run without ever sending tenant data to a third party.

2. Periodic snapshots → drift report

A scheduled task that runs VerseOps.App --refresh --headless once a day, writes the SQLite snapshot to a versioned folder, and emails a delta. "Today vs yesterday: +12 canvas apps in the Default env, –3 envs decommissioned, capacity climbed 4.1 GB on org-prod-eu."

3. Multi-tenant fan-out for MSPs / consultancies

Same EXE, multiple tenant profiles, side-by-side comparison view. The auth layer already supports --tenant <guid>; the cache schema is per-tenant-keyed.

4. Plug-ins for the Inventory API custom queries

The Inventory API's POST /resourcequery/resources/query accepts arbitrary KQL-style projections. A plug-in directory of "common admin questions as queries" (orphaned apps, oldest unused flows, premium connector usage by env) could grow organically.

5. Sister tools in Python / TypeScript

The Python SDK is GA; a Jupyter notebook that mirrors VerseOps' three core panels (env list + capacity + assets) would be ~200 lines and would land instantly with the data-science crowd.

The repository is intended as a working base for these explorations: the foundational ~80% — authentication, caching, paging, retry, redaction, error capture, and theming — is already implemented and exercised against a live ~700-environment tenant. Proposals for any of the directions above can be filed on the issue tracker.

What's in the repository

Everything below is on main at github.com/SweetsNSavories/VerseOps, MIT-licensed:

The single WPF EXE — VerseOps.App/
API clients, one per Microsoft service — VerseOps.App/Inventory/Services/
SQLite catalog schema — VerseOps.App/Inventory/Sql/schema.sql
README.md — install, run, build
SECURITY.md — disclosure policy + threat model
SIGNING.md — three publish-with-signature paths (self-signed dev, Azure Trusted Signing, OV/EV)
docs/network-endpoints.md — every outbound host + OAuth scope
THIRD-PARTY-NOTICES.md + sbom.cdx.json — full dependency attribution + CycloneDX SBOM
CI: build, vulnerability scan, CodeQL — .github/workflows/
Branch protection ruleset (PR required, force-push blocked) — .github/branch-protection.json

Try it

git clone https://github.com/SweetsNSavories/VerseOps.git cd VerseOps dotnet build VerseOps.sln -c Release .\VerseOps.App\bin\Release\net10.0-windows\VerseOps.App.exe

Sign in with a tenant admin account (Power Platform Administrator or Dynamics 365 Administrator), click Refresh, and the first cold pull populates the local cache. Subsequent launches are instant from the cache; click Refresh again whenever you want a fresh snapshot.

Closing thought

The thesis behind this post is straightforward: an API-first Power Platform management surface puts a complete tenant inventory within reach in roughly 3,000 lines of C#. The official Inventory and Usage pages remain the right tool for daily-driver scenarios. The SDK and Inventory API together cover the long tail — the cases where an organization needs a specific join, requires offline operation, or needs the answer the same week.

VerseOps is offered to the community as that starting point. Issues and pull requests are welcome on the public tracker; security disclosures should follow the process documented in SECURITY.md.

— Praveen T · maintainer, VerseOps

References

Power Platform inventory — the in-product surface VerseOps complements
Power Platform admin center Usage page
Programmability and extensibility overview — official tooling map
Versioning and support — the BAP-vs-PPAC story
Programmability — What's new or changed — monthly release log
Power Platform API REST reference (latest)
Microsoft.PowerPlatform.Management on NuGet — the .NET SDK VerseOps consumes
Power Platform for Admins V2 connector — the no-code path to the same API
Tutorial: Create a daily capacity report — Microsoft's own end-to-end SDK example
Power Platform URLs and IP address ranges — for network allow-lists

Archiving Years of Dataverse Audit History Before You Prune It — A Pragmatic, Open Source Pattern

PravinT — Mon, 08 Jun 2026 13:53:50 GMT

Why the audit table is special

The audit table is different from the rest of Dataverse in two ways that matter for an archive design:

It’s an immutable, append-only record of who changed what, when, and from where — the closest thing Dataverse has to a forensic ledger.
The valuable part of an audit row is not the row itself; it’s the diff (old value → new value, attribute mask, related-record context). The audit row stores that diff in a packed changedata column, and the bound RetrieveAuditDetails function is what decodes it into a structured OldValue / NewValue / ChangedAttributes shape your downstream tools can actually query. Synapse Link with the Delta Lake profile will carry the changedata column to the lake, but you still need a parser on the other side; this pattern calls RetrieveAuditDetails at archive-time so what lands in the destination is already decoded and immediately queryable.

That combination makes the audit table the single most useful Dataverse table for:

Regulatory and compliance investigations
“Why did this opportunity status change in Q3 of 2022?” forensic queries (years after the fact)
Internal analytics on user behaviour and process adoption

It’s also the table that grows the fastest. The Dataverse default retention is 90 days, but in practice many enterprises extend that to several years — or set it to never delete — to retain evidence for compliance and forensic review. The result, often after five to seven years, is an audit table holding tens of GB to multiple TB of capacity, dominating the entitlement bill, and rarely accessed in normal operations.

At that point the storage conversation becomes unavoidable. The realistic choices are:

Keep buying entitlement. Predictable, but unbounded.
Move the cold tail somewhere cheaper that you control, then let Dataverse’s audit-deletion job reclaim the space. The hot months stay in Dataverse where users expect them; the years of historical evidence live in your own storage account, queryable when you need them.

This pattern is for option 2 — specifically, for the one-time bulk export of multi-year history, with the option to keep a slow trickle running afterwards if you want to top up.

A crucial point that often gets lost: this pipeline does not need to run live. It is perfectly reasonable to be deliberately months or years behind real-time. The goal is to get a defensible copy of cold data out — the rows you are about to allow Dataverse to delete — not to mirror the audit feed in real time.

What “good” looks like for an external audit copy

Before showing any code, here’s the rubric I held this design to. If you build your own, hold yours to the same rubric:

Property	Why it matters
Idempotent	Re-running the same time window must not duplicate rows. Network blips happen.
Crash-safe (exactly-once-effective)	If the process dies mid-window, the next run must replay the same window cleanly. The watermark advances only after the data is durable in the destination.
Bounded memory	A backlog of millions of audits cannot be loaded all at once.
Backpressure-aware	Dataverse rate-limits aggressively. Throttle responses must not drop rows.
Observable	Every window logs [entity] mode=BACKLOG/LIVE, lag=Nmin, window=10min, records=N so you can watch it work.
Sink-agnostic	The “where does it land” decision is config, not code. Storage choices change; the orchestrator shouldn’t.
Field-level discretion	Audit details can carry PII. The pattern should let admins narrow which attributes leave the platform.

The pattern

The pipeline is conceptually four stages, repeated per entity, per time window:

Three details in this picture do most of the resiliency work. They are deceptively simple:

Detail 1: Half-open time windows (ge / lt)

The boundary moment (09:10:00.000) belongs to window 2, not window 1. So adjacent windows never overlap and never gap, no matter how many times you replay. This is the same trick Kafka uses for offsets — it’s why you can run the loop with confidence.

Detail 2: The destination document key is the Dataverse auditid GUID

Dataverse already assigns a globally unique GUID to every audit row. That GUID becomes the document id in the sink. So when you upsert the same audit twice, the second write is a no-op overwrite of the first — idempotency for free, no client-side dedupe table to maintain.

Detail 3: The watermark moves after the write, not with it

The naive version of this pipeline does:

The resilient version raises a typed exception when any record fails, and the watermark update is conditional on a clean write:

This single change is the difference between “best effort” and “exactly-once-effective.” It’s also the mistake most often made when people roll their own.

Choosing where it lands

The orchestrator is sink-agnostic — it talks to a single AuditSink interface (get_state, update_state, write_audits) and the destination is a config switch, not a code change. The reference implementation ships with four production-shaped sinks plus a no-op for testing. None of them is the answer; they map to platforms enterprises already operate:

Sink	When to consider it
Azure Cosmos DB (NoSQL API)	Operational lookups — “show me everything user X did to record Y in 2022” in milliseconds. Hierarchical partition keys (/entity + /auditYearMonth) keep partitions small as the archive grows over years. Document TTL doubles as a retention policy if you want one. Serverless mode suits a slow-trickle archive workload.
Azure Data Lake Storage Gen2 (Parquet)	The cheap-cold-storage option. Years of audit history land as partitioned Parquet files (entity=…/year=…/month=…/), readable from Fabric notebooks, Synapse Serverless SQL, Databricks, or any Parquet engine. Costs scale with bytes, not throughput — ideal when the archive is rarely queried but must exist.
OneLake (Parquet)	Same Parquet shape as ADLS, but landed inside a Microsoft Fabric Lakehouse. Immediately queryable from a Fabric SQL endpoint, notebooks, and Power BI without further plumbing. The natural choice if your downstream BI is Fabric.
Snowflake (MERGE INTO)	The natural choice when Snowflake is already the analytics platform of record and adding a separate Microsoft analytics estate just for audit data isn’t on the table. MERGE INTO ... ON audit_id keeps the same idempotency contract as the Cosmos upsert, and the warehouse stays paused between archival batches.
No-op (logs only)	First-day connectivity testing. Confirms the Dataverse side works before you provision any storage.

A reasonable default split many enterprises arrive at:

ADLS Gen2 / OneLake (or Snowflake, if that’s your platform) holds the durable historical archive — cheap, partitioned, queryable when (rarely) needed.
Cosmos DB holds the most recent N months for fast operational lookup if there is a use case for it; otherwise skip it entirely.

Adding a sink for storage you already own (e.g., BigQuery, Redshift, on-prem object storage) is roughly 100 lines of Python and one factory entry.

What real-world numbers will look like

The worked test run above shows what a clean sandbox run looks like. Real numbers in your tenant will vary by orders of magnitude depending on:

How many entities have auditing enabled
The shape of RetrieveAuditDetails calls (more changed attributes per row = more bytes per call)
Dataverse Web API rate limits applicable to your environment
Concurrency you allow (max_concurrent_entities in the config)
Sink throughput (Cosmos serverless RU autoscale, ADLS upload bandwidth, Snowflake warehouse size)

The useful operational signal is not the absolute throughput — it’s that the throughput is stable and the per-window log lines tick predictably. If they don’t, look at lag, sink errors, or 429 responses from Dataverse before scaling up concurrency.

When to reach for it (and when not to)

Reasonable fit:

You have multiple years of accumulated audit history in Dataverse and need to move the cold tail off the platform before pruning to reclaim entitlement.
You want the decoded RetrieveAuditDetails payload (old value → new value, attribute mask, related-record context) landing in the destination ready to query — rather than the packed changedata column Synapse Link delivers, which still needs a parser on the consumer side.
Your analytics platform of record sits outside Azure (most commonly Snowflake) and you don’t want to add Synapse + ADLS + Spark to your stack just to land the audit table.
Synapse Link isn’t an option in your tenant — region pairing, governance review, or the cost floor of running ADLS + a Spark pool 24/7 don’t fit your environment.
You’re comfortable running this as a batch job — once for the historical backfill, then perhaps quarterly or annually to top up — rather than as a live continuous feed. Being deliberately months or years behind real-time is fine and often desirable.
You want field-level control over which attributes leave the platform — useful when audit details contain regulated data.

Not a good fit:

You can run Azure Synapse Link with the Delta Lake profile and you’re happy parsing the packed changedata column on the consumer side, and your destination is ADLS / Synapse / Power BI. That’s the supported, first-class path for the audit table — use it.
You only need current state of business tables (account, contact, opportunity). Use Synapse Link / Fabric Link — they do exactly that and you don’t need this pattern.
You need sub-second freshness in the destination. The pattern’s natural cadence is one window length (10 min in the reference config); for true real-time, use Dataverse webhooks or change-tracking APIs.
You don’t have somewhere to operate a small Python container, function, or scheduled job — even an annual one.
You don’t have an internal owner who can be paged when the schedule fails.

The reference implementation

The code that backs this post lives at https://github.com/SweetsNSavories/DataverseAuditLogSyn under MIT, with no warranty. The unified-deployment folder is the version this post describes — single Python codebase, swap sinks via config.json, runs locally / in a container / as an Azure Function.

If you want the implementation depth this post deliberately leaves out — exact API shapes, watermark math, partial-failure drill, sink-author checklist, hosting variants, observability hooks, and the full list of operational responsibilities a self-hosted export carries — it all lives in one place:

unified-deployment/DESIGN.md

Issues, forks, and pull requests welcome via the repo.

Generated 2026-05-14 from https://github.com/SweetsNSavories/DataverseAuditLogSyn/blob/main/docs/blog/archiving-dataverse-audit-logs.md · MIT licensed reference implementation, no warranty.

Azure OpenAI Architecture: The Decisions That Actually Matter (Part 2)

luciacasucci — Mon, 08 Jun 2026 13:53:24 GMT

Introduction

In Part 1 of this series, we walked through the architectural decisions that shape any Azure OpenAI / Microsoft Foundry Models workload: capacity model, deployment location, governance layer, grounding strategy, and quota engineering. Part 2 moves from decisions to discipline. Once you have made those choices, how do you make sure your design holds up to the Azure Well-Architected Framework (WAF)?

The five WAF pillars — Cost Optimization, Security, Reliability, Performance Efficiency, and Operational Excellence — apply just as strongly to GenAI systems as they do to traditional cloud workloads. In fact, they matter more, because GenAI systems are not static: models are upgraded and deprecated, quotas shift, usage patterns grow unpredictably, and new capacity tiers (such as Priority Processing) are introduced while you are running in production.

This post walks through each pillar in the context of Azure OpenAI in Microsoft Foundry, with best practices, common pitfalls, and the trade-offs Cloud Solution Architects (CSAs) tend to hit in real engagements. Where details are time-sensitive — pricing percentages, SLA windows, model retirement timelines, regional rollout delays — they are flagged with "At the time of writing". Always confirm current behavior against Microsoft Learn before committing to a design.

Who is this series for?

Cloud and Solution Architects
Platform and product owners
Senior developers responsible for operating Azure OpenAI workloads in production

What you’ll learn in Part 2:

How each WAF pillar maps to concrete Azure OpenAI design choices.
Where Priority Processing fits across cost and performance trade-offs (and what its eligibility constraints are).
How to plan for model lifecycle events — upgrades, deprecations, retirements — without firefighting.
Which signals to monitor day-to-day, and how to bake them into a GenAIOps loop.
A WAF Decision Matrix at the end of the article, to use as a reusable checklist.

In Part 3, we will look at the part that makes GenAI architecture genuinely different from a traditional service: the platform itself never stops moving.

We’ve also included a summary decision matrix at the end of this post for quick reference.

1. Cost Optimization: Designing for Sustainable Scale

Cost optimization in GenAI is primarily a capacity strategy problem, not just a token-pricing problem. The first question is whether to use pay-as-you-go capacity, reserved capacity, or one of the newer tiers in between.

Reserved capacity (Provisioned Throughput Units, PTUs)

If your workload is steady or growing predictably, you can significantly reduce costs by reserving capacity up front for 1 or 3 years. At the time of writing, reservations typically yield in the range of 30–50% savings compared to hourly pay-as-you-go rates — but the exact discount depends on term length, region, and the model family, so always confirm against the current Azure pricing page.

Fully utilizing a reserved (provisioned) deployment turns cloud spend into a predictable infrastructure investment, much like allocating VM or database capacity. This requires a mindset shift: treat a provisioned Azure OpenAI deployment as always-on infrastructure sized for peak demand, not as on-demand burst capacity.

Importantly, PTU quota is now model-agnostic within a region. You purchase generic throughput units that can be applied to any supported model in that region, so you do not risk stranded capacity when upgrading (say, from one GPT-4 family version to a newer one) or changing model versions. Your investment follows your architecture, not a specific model endpoint.

Avoid dynamic PTU "auto-scaling"

Unlike VM-based infrastructure, dynamically scaling Azure OpenAI capacity up and down to chase cost savings is not recommended. Additional capacity is not guaranteed to be instantly available when you need to scale up, especially if other tenants are consuming the region's resources. Frequent resizing can also negate the benefits of reservations and introduce performance variability. Unused PTUs are not waste — they are headroom that absorbs burst traffic. In practice, design for the peak load and optimize through reservations rather than trying to constantly dial capacity up and down.

Batch tier

Use Batch deployments for asynchronous, non-user-facing jobs (large-scale document processing, nightly data enrichment, evals, embeddings backfills). At the time of writing, Batch can reduce cost per token by up to around 50% compared to Standard pay-as-you-go calls, in exchange for a 24-hour completion window. It also takes pressure off your real-time deployments.

Priority Processing

For workloads that need prompt responses but do not yet warrant a full dedicated PTU deployment, Azure OpenAI offers Priority Processing. Functionally, it is pay-per-token like Standard, but with SLA-backed lower and more consistent latency on the shared infrastructure.

Activation: set the service_tier attribute on the request to "priority" (other values are "default" and "auto").
Model eligibility: at the time of writing, requires model versions released on or after 2025-12-01.
Deployment eligibility: only available on Global Standard or Data Zone Standard (US) deployments.
Pricing: at the time of writing, roughly 20–40% higher per-token cost than Standard, but still meaningfully cheaper than reserving PTU for a low-volume latency-critical path.

Treat Priority Processing as the natural in-between rung: more predictable than Standard for latency-sensitive production traffic, but without the commitment and capacity-planning effort of PTU.

Putting it together

Segment your workloads by interaction pattern and performance need, then assign the most cost-efficient capacity model to each. A common anti-pattern is over-provisioning expensive real-time capacity for jobs that could run asynchronously. Evaluate whether each use case truly requires sub-second latency, or whether a longer batch window (minutes or hours) is acceptable. Use real-time capacity for customer-facing queries and time-sensitive tasks; use Batch or Priority Processing for everything else, depending on tolerance for latency.

[Diagram 1 — Cost Strategy Layering]

2. Security: Compliance, Isolation, and Data Protection

Security in Azure OpenAI begins with deciding where your inference runs and how data is handled. This is often a compliance-driven decision before it is an architectural one.

Deployment scope

Global deployments — Maximize model availability and capacity by allowing Azure to route inference across regions. Pro: broad elasticity and access to the latest models. Con: data is not confined to a single geography, which may violate strict data residency requirements. Global also adds slight troubleshooting complexity, since requests can be served from various regions.
Data Zone deployments — Constrain inference to a specific zone or political boundary (for example, EU-only or US-only Data Zones). Pro: a compliance-friendly middle ground — data processing stays within a defined region set (for example, entirely within the EU to satisfy GDPR), while retaining more elasticity than a single region. Con: slightly reduced model availability and capacity headroom compared to Global.
Regional deployments — Confine inference to one Azure region. Pro: meets the most stringent data sovereignty requirements and can minimize latency for users in that region. Con: limited to the capacity and models available in one region, with no automatic overflow if the region is saturated. New model versions may also roll out to some regions later than others — at the time of writing we have observed delays of roughly 2–6 months for certain releases in specific regions; check Microsoft Learn for the current rollout schedule.

Choosing among these is a regulatory risk-management decision, not just an infrastructure preference. Engage your compliance and data governance teams early to determine the minimum scope of data movement that satisfies requirements. Many teams initially over-constrain this choice out of caution; it is often better to start with a broader deployment (Global or multi-region Data Zone) where permissible, and tighten the scope later if needed. Conversely, if your organization mandates that all data stay in-country, you might go straight to Regional and invest in architectural mitigation for its limitations (capacity planning, multi-region backup plans).

Baseline protections + defense in depth

Regardless of deployment type, Azure OpenAI provides baseline protections: it does not use your prompts or completions to train the underlying models, and all data is encrypted in transit (TLS 1.2+) and at rest (AES-256). Defense in depth is still essential — implement compensating controls at multiple layers:

Redact sensitive data from prompts (or prevent it from being entered) at the client or gateway layer. Use Azure API Management policies or custom middleware to strip out PII or secrets before requests reach the model.
Apply content filtering to both prompts and responses. Use the built-in content filters and/or Azure AI Content Safety to detect and block sensitive or undesirable input and output.
Use strong authentication and role-based access control. Front your Azure OpenAI endpoint with Microsoft Entra ID; scope tokens with least privilege (for example, the Cognitive Services OpenAI User role or managed identity access) instead of distributing master API keys. If a credential is compromised, the blast radius is limited.

Additional best practices

Managed Identities — use them for any internal communication between your application and Azure OpenAI (or other Azure services like storage and databases) instead of embedding API keys. This eliminates the risk of leaking secrets and simplifies credential rotation.
Private endpoints — enable Azure Private Link to keep traffic between your application and the Azure OpenAI service inside your virtual network and the Azure backbone, off the public internet.
Content Safety tooling — integrate Azure AI Content Safety or custom validation functions to scan prompts and completions for policy violations or confidential data. This extra inspection layer can catch issues the base filter misses, and lets you log or modify disallowed content before it reaches the user.

In short, security for GenAI is not just about encryption or API keys — it is about reducing the blast radius of any potential breach or misuse. Confine inference to approved locations, strip sensitive data before it reaches the model, and strictly limit which identities and networks can call your endpoints.

[Diagram 2 — Data Boundary Visualization]

3. Reliability: Designing for Change, Not Just Stability

Reliability in Azure OpenAI is as much about managing model evolution as it is about traditional uptime. Unlike static services, GenAI models are periodically updated and improved by the provider. New versions are released, older versions are deprecated and eventually retired — so a truly reliable system must plan for these changes just as carefully as it plans for hardware failures.

Model lifecycle

At the time of writing, Generally Available (GA) models are typically supported for at least 12 months after release, followed by a deprecation phase of roughly 6 months before retirement. Always confirm the current support windows on Microsoft Learn before locking in a design — these timelines have shifted in the past and may shift again as new model families ship.

When retirement hits:

Standard deployments still pinned to a retired model (with "No Auto-Upgrade" set) stop responding to requests entirely — the API typically returns HTTP 404 (or a similar error) for that model name.
Provisioned deployments using a retired model return HTTP 410 (Gone) errors until you manually switch them to a supported model.

In short, every model version you deploy has a built-in expiration date. Good reliability planning means never being caught unprepared by a model retirement.

Auto-upgrade modes for Standard deployments

Three modes are available:

Auto-upgrade to the latest version — the deployment moves to the new default model version as soon as Azure makes it available. Always on a supported version, but you have no control over timing. Generally not recommended for mission-critical production workloads, since new versions can have different behavior.
Upgrade only on retirement — the deployment stays on its current version until that version is about to be retired, then automatically switches to the latest. Recommended for most production Standard deployments: stability during the model's supported lifespan, with continuity guaranteed at retirement. You still need to test and adjust to the new version, but at least you do not face an outage if you miss the date.
No auto-upgrade — the deployment stays pinned to a specific version unless you change it manually. Not recommended for production: it puts the entire burden on you to track retirement timelines.

Most teams choose option 2 ("upgrade on retirement") for Standard. It allows controlled change during the model's supported period and provides a safety net at retirement. Proactively evaluate new versions for quality, performance, and cost before the forced swap, but the setting greatly reduces the risk of surprise outages.

Provisioned (PTU) migrations

Provisioned deployments do not support auto-upgrade — you must manage these migrations yourself. Azure sends retirement announcements via Azure Service Health alerts and emails, at the time of writing typically 60 days or more in advance. Have a runbook ready. Two approaches are common:

In-place migration — upgrade the deployment's model version through the portal or CLI. The endpoint stays the same and the model is updated behind it. Fast, no new connection string, but expect a brief disruption during the switch and rollback is not straightforward (you may need to contact support to re-enable the old version, if at all possible).
Side-by-side (blue/green) — create a new deployment with the new model version in parallel. Gradually shift traffic (for example, 10% via APIM routing rules), monitor, and roll back instantly if needed. Maximum control and safety, at the cost of running two deployments in parallel for the migration window.

Before any model migration, verify you have sufficient PTU quota in the region for the new model. More advanced models may require more throughput units for the same workload than smaller predecessors — at the time of writing, plan for the possibility that a new generation needs roughly two times (or more) the PTUs to deliver similar throughput. Request quota increases before you hit the upgrade window, not during it.

Multi-region strategy

Consider a multi-region strategy to improve reliability during model rollouts and deprecations. New model versions do not always appear in all regions simultaneously — at the time of writing, Microsoft often launches a model in one region (frequently East US or West Europe) first.

Maintain a secondary deployment in a "first-wave" region to evaluate new versions early.
Use a traffic manager (Azure Front Door, Traffic Manager) to fail over to a region where the model is still available if your primary region lags behind.
Multi-region active-active designs also protect you against single-region outages.

In essence, reliability for GenAI means designing for change. A highly reliable platform is not one that never changes; it is one that changes gracefully. Model upgrades, deprecations, and capacity adjustments should be routine, well-rehearsed events — not fire drills. Achieving this typically requires automation for detecting and applying updates, redundant deployments or regions for flexibility, and ongoing testing of new models well before your current ones retire.

[Diagram 3 — Model Upgrade Strategy]

4. Performance Efficiency: Predictability Over Raw Speed

Performance in GenAI is multi-dimensional. It is not just about raw throughput or the fastest response on an empty system — it is about consistent, predictable latency at scale. Users care that responses are reliably snappy under load, not just fast in ideal conditions.

Performance profiles by capacity model

Standard (shared infrastructure) — multi-tenant, no guaranteed latency SLA. Performance fluctuates with regional demand; you may see throttling (HTTP 429) at peak. Best-effort: great for development, testing, and non-critical workloads, but not a fit for consistent low latency under spikes.
Priority Processing — also shared infrastructure, but your requests jump the queue ahead of Standard traffic. At the time of writing, this is the only pay-per-token tier with an SLA on latency. Activated by setting the service_tier attribute to "priority" on each request (other values: "default", "auto"). Requires model versions released on or after 2025-12-01 and is only available on Global Standard or Data Zone Standard (US) deployments. Pricing premium is roughly 20–40% over Standard. The natural fit for latency-sensitive workloads at intermediate scale — better than Standard, without committing to PTU.
Provisioned Throughput (dedicated capacity) — reserved capacity with isolation from other tenants. The most consistent performance and the strongest Azure SLA on latency (typically bounded p50 and p99 within your provisioned capacity). If your application has strict response-time requirements or user-facing SLAs and the volume justifies it, PTU is the right answer.

A practical pattern: Standard for early-stage and non-critical scenarios; Priority Processing for latency-sensitive paths that have not yet earned a PTU reservation; PTU for steady, high-volume, latency-critical production traffic.

Model selection and configuration

Model size — smaller models generally respond faster than larger ones. Do not automatically pick the biggest model if a smaller one meets your quality bar.
max_tokens — capping response length caps worst-case latency and cost. A 500-token cap finishes sooner than 2000 tokens, even when users ask open-ended questions.
Sampling parameters — low temperature (more deterministic) and a high top_p can shave a small amount of processing overhead versus highly creative or multi-sample setups. Minor compared to model size and length, but real.
Streaming responses — enable streaming wherever possible. The first tokens arrive immediately while the model is still generating; perceived latency drops dramatically even when total time is unchanged.

Treat performance as an explicit design goal. Choose the right capacity model for the job, tune model settings to avoid unnecessary slowdowns, and do not over-engineer with a larger model than needed. A common mistake is defaulting to the biggest model "just in case". Benchmark — a smaller model with good prompt engineering often delivers a fraction of the latency at a fraction of the cost, with negligible quality loss.

5. Operational Excellence: Running GenAI as a Living System

Operational excellence in GenAI means treating your platform as a continuously evolving product. Models change, user behavior shifts, new features ship. Success requires ongoing monitoring, maintenance, and improvement processes — often called GenAIOps (or MLOps for generative AI).

Proactive monitoring

Set up Azure Service Health alerts for your Azure OpenAI / Foundry resource to be notified about service incidents and, importantly, upcoming model deprecations or retirements. At the time of writing, Microsoft typically gives around 60 days of notice for retirement events — but it is easy to miss those notifications if no one is watching. Early awareness lets you test new models and plan migrations calmly instead of reacting at the last minute.

Continuously track key metrics in Azure Monitor or Application Insights, with alerts on anomalies:

Latency percentiles — monitor p50, p95, and p99. A trend up in tail latency is an early warning of saturation or regression.
Error rates — watch HTTP 429 (throttling) and HTTP 503 (server) error trends. Spikes signal capacity limits or service-side issues.
Capacity utilization — for PTU, watch utilization continuously. Sustained operation near 100% means no headroom for bursts. On Standard, watch token usage against subscription limits and quotas.
Token consumption trends — track growth over time. Helps with cost forecasting and reveals runaway usage (unexpectedly popular features, looping clients).

Useful alerting practices: alert on p99 latency breaching a threshold, on any sustained increase in 429s, or when PTU utilization regularly exceeds around 80%. Early warning lets you scale up, optimize, adjust prompts, or throttle specific users before user experience suffers.

Evaluation and reproducibility

Use the evaluation tooling in Azure AI Foundry to compare outputs from two models side by side on a fixed set of test prompts. Re-evaluate periodically — slowly degrading quality often goes unnoticed without a structured comparison.

Implement Infrastructure-as-Code (IaC) and GitOps practices for your Azure OpenAI and supporting resources (APIM, storage, key vault, monitoring). Bicep, ARM, or Terraform templates checked into source control make environments reproducible across dev/test/prod, simplify recovery, and enable change tracking. If something breaks, you can roll back to a known-good configuration quickly.

In summary, operational excellence for GenAI is about continuous learning and improvement. Embrace an AI DevOps culture: invest in monitoring, train your team on model changes, keep optimizing prompts and configurations, and refine processes after each lesson learned. The effort pays off by preventing fire-drills and keeping the platform robust as it evolves.

Final Perspective and Key Takeaways

Applying the Well-Architected Framework to Azure OpenAI forces a higher level of architectural rigor — exactly what GenAI projects in production need. Each pillar drives concrete decisions.

Key takeaways from Part 2:

Cost Optimization — align capacity to workload patterns. Reserve for steady, predictable load; use Batch for offline jobs; use Priority Processing for latency-sensitive paths that do not yet justify PTU; do not pay for ultra-low latency you do not need.
Security — match deployment scope (Global, Data Zone, Regional) to compliance requirements, then layer controls (network isolation, identity and access, data sanitization, content filtering) to minimize blast radius.
Reliability — anticipate continuous model evolution. Use upgrade-on-retirement for Standard, run parallel deployments for PTU migrations, and design for multi-region failover where it matters. Reliability is about avoiding surprises, not just outages.
Performance Efficiency — choose the right capacity model, right-size models and responses, and use streaming. A smaller model with good prompt engineering often beats a bigger one on user experience.
Operational Excellence — treat the platform as a living product. Monitor, alert, automate, evaluate, and version everything as code. The discipline keeps the platform improving instead of decaying.

The organizations that succeed with Azure OpenAI / Microsoft Foundry Models are those that treat capacity planning, security and compliance, model lifecycle management, and governance as first-class design concerns — not afterthoughts. Generative AI architecture is not about deploying a model and walking away; it is about building a resilient, adaptable platform that gracefully evolves as models change and usage grows.

In Part 3, we bring everything together into a comprehensive reference architecture for an enterprise-grade Azure OpenAI platform — combining scalable capacity strategies, layered security and governance, proactive lifecycle (GenAIOps) practices, and multi-region resiliency into a cohesive blueprint ready for production.

[Diagram 4 — WAF pillars summary]

WAF Decision Matrix : Quick Reference

Use this as a checklist when reviewing or sign-off-ing an Azure OpenAI / Microsoft Foundry Models design. One row per decision; one rule of thumb per row.

Pillar	Decision	Rule of thumb	Watch out for
	Capacity tier mix	Variable load → Standard. Latency-critical → Priority Processing. Offline bulk → Batch. Steady high-volume → PTU.	Single-tier platforms over-pay for elasticity or under-deliver on latency.
Cost	Reservation term (PTU)	1- or 3-year terms for predictable workloads; size for peak, not for average.	Dynamic resizing of PTU to chase savings; capacity not guaranteed at scale-up time.
Cost	Priority Processing eligibility	Requires service_tier="priority", model 2025-12-01+, Global Standard or Data Zone Standard (US).	Assuming it works on every region/model — confirm eligibility before committing the design.
Security	Deployment scope	No residency rule → Global. Multi-region zone OK → Data Zone. Strict residency → Regional.	Over-constraining out of caution; or under-constraining and missing a compliance requirement.
Security	Identity and access	Microsoft Entra ID + Managed Identity + scoped roles. No master keys in apps.	Long-lived API keys distributed across teams.
Security	Network and data	Private Link for in-network traffic; PII redaction at the gateway; content filtering on prompts and responses.	Public endpoints, raw PII in prompts, only relying on the built-in filter.
Reliability	Auto-upgrade strategy	Standard → "upgrade on retirement". PTU → planned blue/green migration with sufficient quota in advance.	Pinning Standard with no auto-upgrade and forgetting; in-place PTU migration with no rollback path.
Reliability	Multi-region	Active-active (or first-wave secondary) for critical paths; traffic manager for failover.	Single-region production with no plan for capacity or model-rollout lag.
Performance	Capacity match	Match capacity tier to latency target: Standard for non-critical; Priority for latency-sensitive; PTU for SLA-bound.	Expecting Standard to deliver consistent low latency under spike load.
Performance	Model and response sizing	Pick the smallest model that meets quality. Cap max_tokens. Stream responses.	Defaulting to the largest model "just in case"; long uncapped responses; no streaming.
Operations	Monitoring and alerting	Track p50/p95/p99, 429/503 rates, PTU utilization, and token trends. Alert on tail latency and sustained throttling.	Average-only dashboards; missed Service Health notifications for model retirements.
Operations	IaC and GitOps	Bicep/ARM/Terraform under source control; reproducible dev/test/prod; pipeline-driven changes.	Click-ops in the portal; environment drift between dev and prod.

Disclaimer

I am a Microsoft employee. The views and opinions expressed in this article are my own and do not necessarily reflect those of Microsoft. This content is informational and educational; it is not an official Microsoft statement, recommendation, or commitment. Service tiers, model availability, pricing, SLAs, and feature eligibility evolve — always validate against the latest Microsoft Learn documentation before making architectural decisions.

References

Azure OpenAI Architecture: The Decisions That Actually Matter (Part 1)

luciacasucci — Mon, 08 Jun 2026 13:53:07 GMT

Generative AI demos often succeed because they hide the hard parts of architecture. They usually run under ideal conditions: low, steady traffic, no sudden bursts, no competing teams, and minimal regulatory scrutiny. In production, however, Azure OpenAI systems face a very different reality – variable loads, service quotas, compliance constraints, evolving model versions, and the need for cost visibility.

The difference between a great demo and a resilient production platform isn’t the model itself – it’s the early architectural decisions. The choices you make from day one determine whether your generative AI solution can handle real-world demand or buckle under pressure.

Who is this series for?

Cloud and Solution Architects
Platform and product owners
Senior developers responsible for operating Azure OpenAI workloads in production

What you’ll learn in Part 1: We’ll walk through five foundational design decisions for Azure OpenAI, explain why they matter, and highlight key trade-offs and pitfalls we’ve seen in real-world deployments:

Capacity Model – Choosing between Standard (PAYGO), Priority Processing, Batch, or Provisioned Throughput (PTU)
Deployment Location – Global vs. Data Zone vs. Regional hosting
Governance Layer – When and why to introduce a GenAI gateway
Grounding Strategy – When to use Retrieval-Augmented Generation (RAG)
Quota Engineering – How to plan for service limits and avoid throttling

In Part 2, we’ll translate these principles into concrete implementations: multi-region topologies, cost allocation strategies, observability and monitoring patterns, and other best practices for reliability, security, and DevOps in Azure OpenAI. Part 3 will connect these decisions to GenAIOps best practices to help ensure your solution is future-proof.

We’ve also included a summary decision matrix at the end of this post for quick reference.

1. Capacity Model: PAYGO, Priority, Batch, or Dedicated Throughput?

At the time of writing, Azure OpenAI (now part of Microsoft Foundry Models) offers four capacity models for hosting models, each with distinct cost, latency, and operational characteristics. Most production solutions combine two or more of these tiers:

Standard (pay-as-you-go, shared): Multi-tenant, elastic capacity. You pay per token, with no upfront commitment and no cost when idle.
Pros: Simple and flexible; ideal for dev/test and for moderate or unpredictable traffic patterns.
Cons: No guaranteed throughput or low-latency SLA – performance may vary with regional load. Under heavy usage, you may see high latency or HTTP 429 “Too Many Requests” errors due to shared capacity limits.
Priority Processing (pay-per-token, SLA-backed low latency): A pay-as-you-go service tier that routes traffic through reserved compute, giving consistent low latency for business-critical, user-facing workloads without buying PTUs. At the time of writing, it is available on Global Standard and Data Zone Standard (US) deployments and requires recent model versions (2025-12-01 or later). It can be enabled per deployment in Microsoft Foundry, or set on individual API calls via the optional service_tier attribute (auto / default / priority) on the chat completions and responses APIs. Always confirm current model and region eligibility in the Microsoft Learn article “Enable priority processing for Microsoft Foundry Models.”
Pros: Predictable, low-latency responses with the simplicity of pay-per-token billing – a strong fit for bursty, business-hour, or latency-sensitive traffic where PTU commitment isn’t justified. Uses the same Standard quota pool, and can be layered on top of PTU for steady-state capacity plus an elastic priority lane for spikes.
Cons: Per-token pricing is higher than Standard PAYGO. Requires eligible deployment types and current model versions, so it is not a drop-in for legacy deployments. Like all PAYGO modes, it is still subject to TPM/RPM limits and the same throttling behaviour if quota is exhausted.
Batch (asynchronous): Offline job processing. You submit requests in bulk (e.g., via a file) and receive results after up to 24 hours.
Pros: Optimized for high throughput at a much lower cost (roughly 50% less per token than real-time requests at the time of writing). Batch jobs use separate “enqueued tokens” quotas, so they won’t interfere with interactive traffic.
Cons: Not suitable for real-time use – no immediate responses or latency guarantees. Requires extra orchestration (staging requests, handling outputs). At the time of writing, Batch does not support embedding models, so vector indexing jobs must use Standard mode – always check the Azure OpenAI documentation for the latest supported model list.
Provisioned Throughput (dedicated PTUs): Reserved, dedicated capacity. You purchase a fixed amount of capacity (Provisioned Throughput Units) and pay for it hourly, whether used or not.
Pros: Guaranteed throughput and consistent low latency, since you’re isolated from other tenants; suitable for high-volume services with strict SLAs.
Cons: Requires careful sizing of PTUs to match your peak demand – under-provision and you’ll still get 429s (now self-inflicted), over-provision and you pay for unused capacity. In addition, output tokens count more heavily against PTU usage (e.g., generative tokens from GPT-4 consume multiple capacity units each), so planning must account for both prompt and completion length.

To right-size your PTU, you’ll need an estimate of the following metrics:

Requests per minute
Average tokens per request
Peak concurrency
Prompt + completion token size

By plugging these into the Azure OpenAI PTU Calculator (linked in the References section), you can get a first estimate of the size of purchase you need to make based on your consumption.

Most production solutions use a hybrid approach: for example, Provisioned capacity for steady, critical real-time traffic, Priority Processing for latency-sensitive bursts that exceed PTU headroom, Standard for overflow or early-stage apps, and Batch for large-scale offline processing. As a rule of thumb: if a user is waiting for a response, use a real-time endpoint (Standard, Priority, or PTU). Use PTU when you need strict latency consistency at high, predictable volume; use Priority Processing when you need SLA-backed low latency without committing to PTUs; use Standard for everything else interactive. If a task can be handled asynchronously, offload it to Batch to reduce cost and keep interactive systems responsive.

Pitfall – Sizing for average load instead of peak burst. One company provisioned capacity only for typical throughput and was overwhelmed when traffic spiked ~3× beyond normal. They maxed out their PTU allocation, triggering a flood of 429 errors. Lesson: model your peak tokens-per-minute (TPM) and requests-per-minute (RPM), not just the average, and add a safety margin to avoid unexpected throttling.

Insight – Separate real-time and background workloads. An initial version of a news analytics bot processed all articles on demand, leading to slow, costly responses. The team later moved heavy processing to Batch jobs (pre-computing article embeddings and summaries), cutting end-user latency by 80% and halving costs.

[Diagram 1 – Capacity Model and Deployment]

2. Deployment Location: Where Does Inference Run?

After choosing the capacity model, decide where your Azure OpenAI instance is hosted. This affects latency, scalability, compliance, and model availability. Azure provides three options:

Global – Your endpoint isn’t tied to a specific region.
Pros: Maximum elasticity and often the best performance stability, since Azure can route traffic to any available regional capacity. You also usually get access to new model releases first on global endpoints.
Cons: Data is processed across multiple regions (may violate strict data residency needs). Troubleshooting can be more complex when calls are served from various locations.
Data Zone – Inference is restricted to a defined geography (e.g., all Azure EU regions).
Pros: Ensures data stays within a specific political boundary for compliance (e.g., GDPR) while retaining some elasticity across multiple regional datacenters in that zone.
Cons: Smaller capacity pool than Global, and possibly a slight delay in getting certain new model versions compared to global rollout. A good balance if you require geographic control without completely sacrificing scalability.
Regional – Inference runs in a single Azure region that you choose.
Pros: Strict data residency and potentially minimal latency if your users are near that region.
Cons: No ability to burst to other regions – you are limited to one datacenter’s capacity. If that region faces high load or an outage, your service is impacted. Some model versions or features may also take longer to become available in a given region than on Global. For current model and region availability, refer to the Azure AI Foundry – Model Deployment Types documentation in the References section.

Pitfall – Over-constraining location without need. Some teams unnecessarily default to a narrow deployment. For instance, a company chose a local Regional deployment out of habit, then discovered the Azure OpenAI model they needed wasn’t available in that region for several months, forcing a last-minute migration to a broader Data Zone. Lesson: unless you have a clear compliance or latency requirement, start with a less restrictive option (Global or multi-region Data Zone) to avoid capacity or availability issues.

Example: One enterprise began with a Global deployment for performance and simplicity, but later had to move to an EU Data Zone to meet GDPR rules, trading some elasticity for compliance. Conversely, a team that started with a single-region setup ran into scaling limits and delayed feature rollouts; they eventually reconfigured to a Data Zone to tap into a larger resource pool.

3. Governance Layer: When a GenAI Gateway Is Needed

As Azure OpenAI usage scales to multiple applications or teams, direct API calls from each app become hard to manage. A central API gateway (such as an Azure API Management instance in front of the OpenAI endpoints) is recommended to enforce enterprise policies and provide a single point of oversight. A gateway enables:

Central Authentication & RBAC: Use Microsoft Entra ID for authentication instead of distributing API keys, and enforce role-based access so each app or team only accesses allowed resources.
Usage Quotas & Throttling: Allocate token or request quotas per application or client. This prevents one service from monopolizing the OpenAI service and can smooth out bursts by applying backpressure (e.g., returning 429s or queueing requests) before Azure OpenAI’s own limits are exceeded.
Intelligent Routing: Direct traffic flexibly – route most requests to a primary model deployment, send a fraction to a new model version (canary), or fail over to a secondary region or the Standard tier if the primary is constrained.
Unified Monitoring & Cost Management: Log all requests in one place. This gives you a clear view of consumption by team or feature, helps with debugging, and supports internal charge-back or cost governance.

A gateway doesn’t make the model faster or more scalable by itself – it’s about control, security, and manageability, not raw performance. That said, for any multi-team or multi-application scenario, a gateway quickly becomes essential to avoid “shadow AI” deployments and chaotic usage patterns.

When to add: Introduce a GenAI gateway once more than one application or team is using the service, or whenever you need to enforce cross-cutting policies. Implementing it early can save a lot of headaches compared to retrofitting it later.

Example: An e-commerce company initially allowed several departments to call the Azure OpenAI API directly. Soon, they had no clear visibility into who was using how many tokens, and costs spiked unexpectedly. They deployed an APIM gateway to require proper authentication, impose per-app quotas, and log usage metrics. The result was rapid identification of the top token-consuming app (preventing it from starving others) and much better cost control.

[Diagram 2 – GenAI Gateway Functionalities and deployment Location sprectrum]

4. Grounding Strategy: When to Use RAG (Retrieval-Augmented Generation)

Many enterprise use cases demand that the model’s answers include specific internal knowledge or citations. Retrieval-Augmented Generation (RAG) is the solution when your AI needs to ground its responses in external data. RAG works by retrieving relevant content from your own data sources and providing it to the model in the prompt:

Document Indexing: First, collect your reference documents (files, knowledge bases, etc.) and break them into chunks, optionally adding metadata (titles, tags). Store these in a vector index or search database after transforming each chunk into an embedding vector.
Relevant Retrieval: For each user query, create an embedding of the question and retrieve the top-matching document chunks from the index via similarity search.
Augmented Prompt: Prepend or append the retrieved text snippets to the model’s prompt (often along with instructions to use them for reference).
LLM Response: The model (e.g., GPT-4) processes the augmented prompt and generates an answer that incorporates the provided reference information (often with source citations if required).

By injecting enterprise data at prompt time, RAG can significantly reduce hallucinations and increase the factual accuracy of outputs. Users get answers that reflect real data you’ve provided, rather than just the model’s training data.

However, RAG adds complexity, latency, and cost. You must maintain additional infrastructure (embedding computation and a vector store or search index). Each query now has extra steps, typically adding 200–500 ms to response time. The vector database and compute for embeddings also incur costs – industry estimates often put a full RAG pipeline at 3–5× the cost of using the base model alone, especially at scale. You’ll need a strategy for keeping your index updated as source data changes, and robust handling for cases where no relevant data is found.

When to use RAG: Use RAG if your model must reliably incorporate proprietary, dynamic, or highly specific information that isn’t part of its training data, or when you need to provide source references for answers. If your scenario is more open-ended or doesn’t require up-to-date factual grounding, you can often skip RAG to keep the system simpler and faster.

Pitfall: Some teams adopt RAG by default, which can slow development and complicate the system unnecessarily. It’s often better to start with a simpler approach and add RAG later if you find the model’s answers need external support.

Examples: One consulting firm added RAG to their internal Q&A bot to leverage proprietary research. Answers became more accurate, but query latency jumped to ~5 seconds due to the retrieval overhead, forcing them to optimize their embeddings pipeline and caching. Conversely, a health company launched a chatbot without RAG and discovered it gave incorrect medical answers because it couldn’t reference the latest policy documents – a failure that a RAG approach could have mitigated.

[Diagram 3 – RAG High Level Flow and Anatomy]

5. Quota Engineering: Avoiding Bottlenecks and Throttling

The Azure OpenAI Service imposes quota limits to protect the system. If you don’t plan for these, they can become points of failure in production. Key limits include:

Tokens per Minute (TPM): Maximum tokens (input + output) your deployment can process per minute (your primary throughput cap).
Requests per Minute (RPM): Maximum number of API calls per minute.
Concurrent Requests: Maximum number of requests processed simultaneously.
Model-specific limits: Certain model types have their own constraints (e.g., the maximum request rate for GPT-4 may be lower than for GPT-3.5 due to higher computational load).

If you exceed these limits, Azure OpenAI will return errors – usually HTTP 429 (Too Many Requests) for quota exhaustion or 503 (Service Unavailable) if the service is stressed. In other words, hitting a quota isn’t a theoretical worry; it will result in rejected requests once you cross the threshold.

Quota Tiers and Deployment Types

Azure OpenAI uses a tiered quota system where limits depend on your subscription’s access level. Specific numeric quotas change frequently, so always verify against the Azure OpenAI Quota Guide (linked in the References section).

Tier 1 (Default): Standard quota allocations suitable for development and moderate production workloads. At the time of writing, GPT-4 deployments in Tier 1 commonly start at quotas in the low tens of thousands of TPM and around a thousand RPM, but exact values vary by region and model – always check the current Azure OpenAI Quota Guide for live numbers.
Tier 2 and Above: Higher quotas available through approval processes, typically for enterprise customers with demonstrated high-volume needs. These tiers can provide significantly more capacity than Tier 1; consult the Quota Guide for current multipliers and approval paths.

Standard (pay-as-you-go) deployments share regional quota pools and are subject to TPM/RPM limits that can vary by region and model. Provisioned Throughput (PTU) deployments operate differently – you purchase dedicated capacity measured in PTUs, and your throughput is determined by your PTU allocation rather than by TPM/RPM limits. PTUs still have implicit rate limits based on the processing capacity of your purchased units.

The Batch API uses a separate quota system with “enqueued tokens” limits, allowing much higher total throughput (often millions of tokens per day) but without real-time guarantees.

Best practices to manage quotas:

Capacity planning: Calculate your peak usage requirements (e.g., max prompt+completion tokens per request × peak requests per minute). Ensure your chosen plan or quota can handle this, or request an increase in advance.
Design for bursts: Traffic often comes in waves. Aim to operate well below your limits so you can absorb sudden surges. As a guideline, keep usage under ~70% of your TPM/RPM limits during normal operation, leaving headroom for peaks. If your usage is spiking above 85% regularly at the 95th percentile, it’s time to scale up capacity or optimize usage.
Graceful degradation: Implement exponential backoff (with jitter) on the client side when retries are necessary. This prevents a stampede of retries (a “retry storm”) that would otherwise compound the load problem. At the platform level, use queues or token-bucket rate limiters (possibly in your APIM gateway) to smooth bursts.
Circuit breakers: Have fallback plans for extreme scenarios. Temporarily disable non-critical features or queue requests when approaching critical limits to prevent a total outage.

Example: A fintech company’s trading chatbot ran fine in testing, but during a market surge their question volume tripled. This breached their tokens-per-minute quota and led to a flood of 429 errors. Worse, their code immediately retried each failed request without delay, intensifying the load and effectively causing a self-inflicted denial-of-service outage. They resolved it by using exponential backoff and partitioning users across multiple deployments.

[Diagram 4 – Quota Engineering]

Final Perspective and Key Takeaways

Ultimately, building a production-grade Azure OpenAI solution is much more about well-structured cloud architecture than about the model itself. An advanced model can underperform in a fragile setup, while even a basic model can excel in a solid architecture.

Key takeaways from Part 1:

Plan for peak loads. Design for the worst-case traffic (and add buffer), not the average. If you need strict performance guarantees, invest in dedicated capacity early.
Avoid unnecessary constraints. Don’t lock into a restricted deployment unless required by compliance or latency – new models and extra capacity reach global deployments first.
Use real-time vs. batch wisely. Real-time endpoints (Standard, Priority, or PTU) should be reserved for interactive, user-facing tasks; move large or non-urgent jobs to Batch for roughly half the cost per token (at the time of writing).
Pick the right real-time tier. Use Priority Processing when you need SLA-backed low latency without committing to PTUs, PTU for high, predictable volumes, and Standard for everything else interactive.
Implement a gateway for scale. If you have multiple applications or teams, use an API Management gateway for authentication, rate limiting, logging, and multi-region routing.
Adopt RAG only if needed. Don’t introduce a retrieval-augmented generation layer unless your application truly demands external data or source citations.
Engineer for limits and failure. Treat rate limits and error handling as fundamental design criteria. Build in monitoring, backoff, and fallback mechanisms so the system degrades gracefully.

In short, succeeding with Azure OpenAI in production means treating it as a full-stack architecture challenge rather than just an API integration. By proactively addressing scalability, deployment, governance, data grounding, and quotas, you can turn a promising demo into a stable, cost-efficient, and compliant AI platform.

In the next part, we’ll explore how to put these principles into practice – including multi-region architectures, cost-sharing strategies for teams, advanced monitoring/logging setups, and other patterns for making Azure OpenAI a robust enterprise service.

Decision Matrix: Quick Reference

Use this matrix as a fast first cut on the five Part 1 decisions. It is not a substitute for a full design review, but it captures the trade-offs most teams need to evaluate up front.

Decision	Choose this when…	Avoid when…	Primary risk
Capacity – Standard (PAYGO)	Dev/test, unpredictable or bursty traffic, MVPs, overflow	You need guaranteed latency or strict SLAs	429 throttling under shared load
Capacity – Priority Processing	Latency-sensitive, business-critical real-time traffic where you don’t want PTU commitment; burst lane on top of PTU	Deployment type / model version isn’t eligible; cost-sensitive, low-priority workloads	Higher per-token cost than Standard; still PAYGO quota-bound
Capacity – Batch	Async jobs, embeddings refresh, large offline summarization	User is waiting; or model is unsupported (e.g., embeddings, at the time of writing)	Up to 24h turnaround; orchestration overhead
Capacity – Provisioned (PTU)	High-volume real-time workloads with strict SLAs and predictable load	Demand is low or highly variable	Over-/under-provisioning costs
Deployment – Global	You want widest model availability and best elasticity	Strict data residency or regulatory constraints	Less control over where data is processed
Deployment – Data Zone	Geographic compliance (e.g., EU/US) with some elasticity	Single-region residency is mandated	Some lag on newest model versions
Deployment – Regional	Strict data residency or co-located low-latency users	You need to scale beyond one region’s capacity	Capacity ceilings and slower model rollout
Governance – GenAI Gateway	Multiple apps/teams, need RBAC, quotas, routing, central logging	Single small app where overhead exceeds benefit	Adds latency and another component to operate
Grounding – RAG	Need proprietary, dynamic, or cited answers	Open-ended creative tasks where freshness isn’t required	Latency, cost, and index freshness drift
Quota – Plan & Tier Up Early	You’re close to TPM/RPM ceilings or expecting growth	Your peak forecast is well below current quota	Last-minute throttling and outages

Tip: If you can only optimize for one decision in Part 1, start with capacity model and quota engineering – they are the two most common sources of production incidents we see in real deployments. Pairing PTUs (or Priority Processing) for steady, latency-sensitive traffic with Standard PAYGO for overflow is a pattern that consistently delivers both reliability and cost control.

Disclaimer

The views expressed in this article are those of the author and do not necessarily reflect the official policy or position of Microsoft. The author is a Microsoft employee.

References

Enabling AI-Driven SAP Development with GitHub Copilot: Plans and Usage-Based Billing Transformation

AnuradhaKarnam — Mon, 08 Jun 2026 13:52:48 GMT

Introduction:

Empower SAP customers with a strategic understanding of GitHub Copilot offerings, including plan options, value realization, account models, and key considerations for enterprise adoption. Through a guided, step-by-step setup in Visual Studio Code, SAP developers will gain hands-on experience with GitHub Copilot and Copilot Chat in real-world development scenarios. In addition, a practical exercise demonstrates how to configure and optimize Copilot within Visual Studio Code enabling SAP teams to effectively adopt AI-assisted development and accelerate application delivery.

Imagine you are an SAP developer working within a fast-paced enterprise environment, tasked with delivering a new feature or extension for a business-critical application, such as SAP S/4HANA or SAP BTP. With limited familiarity with the existing codebase and tight delivery timelines, ensuring quality, performance, and reliability is essential. GitHub Copilot provides a powerful AI-assisted development experience that can help SAP developers accelerate delivery while maintaining high standards. It can interpret complex code, generate documentation, and assist in writing efficient, error-free code enabling teams to meet deadlines and deliver high-quality solutions with confidence.

GitHub Copilot plans for SAP Customers:

Starting June 1, 2026, GitHub is transitioning GitHub Copilot from a request-based billing model to a usage-based billing model. For SAP customers, this shift enables more granular cost alignment with actual AI usage across development teams, supporting improved transparency and budget control within SAP programs.

Organizations and enterprises can leverage this model to better manage consumption across SAP development scenarios such as S/4HANA extensions, BTP applications, and integrations while optimizing costs based on real usage patterns. Individual SAP developers will also benefit from flexible billing aligned to their actual Copilot usage.

GitHub Copilot Capabilities for SAP Development:

GitHub Copilot offers a range of capabilities designed to support SAP developers across the entire development lifecycle from coding and testing to collaboration and deployment.

GitHub Copilot in the CLI: SAP developers can use a chat-like interface directly in the terminal to interact with command-line tools. Copilot provides command suggestions and explanations, helping streamline development workflows for SAP integrations, deployments, and automation tasks.

AI-Generated Pull Request Summaries: Copilot generates intelligent summaries of pull requests, highlighting key changes, impacted files, and areas requiring attention. This enables SAP teams to accelerate code reviews and improve collaboration especially in complex enterprise environments. (Not available in the Free plan.)

GitHub Copilot Extensions
Copilot Extensions allow SAP customers to integrate external tools and systems such as SAP BTP services or other enterprise platforms into the development workflow. These extensions can be customized and shared to support organization-specific SAP scenarios.

Custom Instructions
SAP teams can tailor Copilot responses by providing context about development standards, tools and project requirements ensuring outputs align with SAP best practices and enterprise guidelines.

GitHub Copilot Memory (Public Preview)
Copilot can learn from and retain context about a repository, improving the relevance and quality of suggestions. This is particularly valuable for SAP projects with complex codebases and domain-specific logic.

GitHub Copilot Spaces
Organize SAP project assets such as code, documentation, and specifications into contextual “Spaces.” This ensures Copilot delivers responses grounded in the appropriate SAP business and technical context.

GitHub Copilot in GitHub Desktop
Automatically generate commit messages and descriptions based on SAP code changes, improving consistency, traceability, and developer efficiency.

GitHub Copilot Agents for SAP Development:

GitHub Copilot Chat in Visual Studio Code provides specialized agents designed to support different stages of the SAP development lifecycle ranging from planning to implementation and knowledge exploration.

Agent (Execution Agent)
Use the Agent when you need to implement a specific SAP development task, such as building extensions, integrations, or enhancements for SAP S/4HANA or SAP BTP.
The Agent can autonomously identify relevant files, propose code changes, execute supporting commands, and iteratively refine outputs to complete the task efficiently.

Plan (Planning Agent)
Use the Plan agent when you want to define a structured, step-by-step approach before implementation. This is particularly useful for SAP scenarios requiring careful design such as complex integrations or multi-system workflows. The Plan agent breaks down requirements into clear steps that can be executed by the Agent.

Ask (Knowledge Agent)
Use the Ask agent to explore SAP-related code, development patterns, or general technical concepts without making changes. This is ideal for understanding unfamiliar codebases, learning new SAP development paradigms, or clarifying architecture and best practices.

Implementing GitHub Copilot in SAP Environments:

SAP organizations can follow a structured approach to successfully set up and scale GitHub Copilot across development teams:

Subscribe and Select the Right Plan
Begin by subscribing to GitHub Copilot and selecting the appropriate plan (Business or Enterprise) based on your SAP landscape, governance needs, and scale of adoption.

Establish Governance and Policies
Define organizational policies to control how GitHub Copilot is used across SAP development scenarios. This includes enabling or restricting specific features to align with compliance, security, and SAP development standards.

Configure Secure Networking
Ensure secure connectivity by configuring proxy servers or firewalls to allow required Copilot endpoints. SAP customers may also need to manage SSL certificates to align with enterprise security requirements.

Provision Access to Development Teams
Enable GitHub Copilot for selected SAP developers or teams such as those working on SAP BTP extensions, integrations, or modernization initiatives. A phased rollout can help identify challenges and demonstrate early value before scaling organization wide.

Drive Adoption Across SAP Teams
Establish a structured enablement strategy including training, best practices, and internal champions to promote effective usage of Copilot within SAP development workflows. A self-service licensing model can further accelerate adoption.

Support Developer Authentication
In enterprise environments, ensure SAP developers complete the required authentication steps to access Copilot from their development tools, such as Visual Studio Code.

Enhance the Copilot Experience
SAP organizations can further optimize value by:

Integrating knowledge bases to provide SAP-specific context (Enterprise plan)

Customizing Copilot through tailored AI models aligned to SAP development needs

Leveraging Copilot Extensions to integrate SAP tools and other enterprise systems into the development workflow

Set Up GitHub Copilot in Visual Studio Code:

For SAP developers, setting up GitHub Copilot is a simple process:

Ensure you are using the latest version of Visual Studio Code to support modern SAP development workflows (e.g., SAP BTP and full-stack development).

In the Status Bar, select the GitHub Copilot icon and choose Use AI Features to begin setup.

If your account is already authenticated, select Set up Copilot to complete activation.

Follow the guided steps to authorize Copilot within your development environment.

If no subscription is assigned, SAP developers will be onboarded to the GitHub Copilot Free plan for initial evaluation.

Once setup is complete, GitHub Copilot is ready to assist with SAP development tasks such as code generation, documentation, and optimization.

Customize GitHub Copilot Settings for SAP Development:

SAP developers can tailor GitHub Copilot behavior to align with enterprise development standards and project requirements:

Access Settings by selecting the gear icon in the lower-left corner of Visual Studio Code.

Navigate to GitHub Copilot configurations, organized into key areas:

Code Editing Settings
Configure inline code suggestions, next-edit predictions, and language-specific behaviors to support SAP development languages and frameworks.

Chat Settings
Customize how Copilot Chat behaves, including terminal integration helpful for managing SAP build, deployment, and integration tasks.

Agent Settings
Control advanced capabilities such as agent mode, request limits, and tool approvals to align with SAP governance, security, and compliance requirements.

Selecting Where GitHub Copilot Agents Run in SAP Development:

In addition to selecting the appropriate Copilot agent for a specific task, SAP developers can control where the agent executes using the Agent Target option in the Chat view. This capability determines how the agent interacts with SAP development environments and when results are delivered.

Local Execution
Runs directly within the developer’s environment with full access to the workspace, tools, and codebase. Ideal for SAP developers working on real-time coding tasks, exploring complex SAP codebases, or making controlled updates where step-by-step review is required.

GitHub Copilot CLI (Local Background Execution)
Executes tasks in the background on the developer’s machine. Suitable for SAP scenarios where predefined tasks such as script generation, automation, or batch updates can run while developers continue working on other SAP components.

Cloud Execution
Runs remotely in GitHub and integrates with workflows such as issues and pull requests. This option is well-suited for SAP teams collaborating on large-scale projects, enabling developers to delegate tasks, generate pull requests, and streamline code reviews across distributed teams.

Third-Party Execution
Uses external AI platforms (such as Anthropic or OpenAI) to run agents either locally or in the cloud. This provides flexibility for SAP customers to integrate advanced AI capabilities into their development workflows, depending on enterprise architecture and governance requirements.

Controlling Agent Permissions (for SAP Customers):

You can define how much autonomy your digital agents have by using the Permissions settings within the Chat experience. Adjusting these levels allows you to balance operational efficiency with governance and oversight an important consideration in SAP-driven business processes.

Permission Level	Description
Default Approval	Only low-risk, read-only actions are executed automatically. Most agent-initiated actions require user confirmation, ensuring control over business-critical operations and data changes.
Bypass Approvals	All agent actions are automatically approved without user prompts. This setting streamlines workflows but should be used with appropriate governance controls, especially in sensitive SAP environments.
Autopilot	Enables fully autonomous execution. The agent handles actions end-to-end, including responding to clarification requests, without user intervention. Ideal for well-defined scenarios, but requires strong trust, monitoring, and compliance alignment.

Conclusion:

GitHub Copilot offers SAP organizations a powerful way to modernize development through AI-assisted engineering. It combines intelligent code generation, contextual insights, and advanced features like agents, CLI integration, and memory to accelerate delivery while maintaining enterprise-grade quality and compliance.

It’s usage-based billing model further enhances value by aligning costs with actual adoption, providing transparency, scalability, and control across scenarios such as S/4HANA extensions, SAP BTP applications, and integrations. With a structured approach, covering plan selection, governance, secure configuration, and phased rollout, organizations can integrate Copilot effectively while meeting SAP-specific standards. Ultimately, Copilot enables SAP teams to boost productivity, enhance collaboration, and drive innovation at scale, delivering faster time to market and sustained business value.

Reference links:

Plans for GitHub Copilot - GitHub Docs

Usage-based billing for individuals - GitHub Docs

Usage-based billing for organizations and enterprises - GitHub Docs

Databricks Lakebase: The operational database for AI agents and apps

anishekkamal — Fri, 01 May 2026 18:23:22 GMT

Understanding the Evolution: From Lakehouse to Lakebase

The modern data landscape has long been characterized by a fundamental schism: Online Transaction Processing (OLTP) systems, designed for high-frequency, low-latency transactions in applications, and Online Analytical Processing (OLAP) systems, optimized for complex queries, reporting, and machine learning on vast datasets. This division historically necessitated intricate and often fragile Extract, Transform, Load (ETL) processes to move and synchronize data between these disparate environments, leading to increased complexity, data duplication, and governance challenges.

Databricks Lakehouse architecture emerged to unify data warehousing and data lake f

unctionalities for analytical workloads, offering the flexibility of data lakes with the performance and governance of data warehouses. However, a critical piece remained: native, high-performance OLTP capabilities directly within this unified environment. This is where Databricks Lakebase enters the picture, representing a significant evolution by bringing fully managed PostgreSQL OLTP capabilities directly into the Databricks Data Intelligence Platform.

Lakebase addresses the need for a single, governed platform that can seamlessly handle both transactional and analytical workloads, thereby simplifying data architectures, reducing operational overhead, and accelerating the development of real-time applications and AI agents. By integrating OLTP at the core of the lakehouse, Databricks aims to create a truly unified data and AI platform.

1.Visualizing the architectural shift: Lakebase integrates seamlessly within the Databricks Lakehouse ecosystem.

The Architectural Innovation: Separation of Compute and Storage

At the heart of Databricks Lakebase's efficiency and scalability lies its innovative architecture, which fundamentally separates compute from storage. Unlike traditional monolithic databases where these components are tightly coupled, Lakebase decouples them, offering distinct advantages:

Elastic Scaling and Cost Efficiency

The transactional compute layer in Lakebase is serverless and ephemeral, meaning it can scale up or down dynamically based on demand. This includes the ability to scale to zero during periods of inactivity, significantly optimizing cost by ensuring you only pay for the compute resources actively used. Data, on the other hand, is persisted directly into low-cost, durable cloud object storage (e.g., Azure Blob Storage) using open formats like Delta Lake. This design not only reduces storage costs but also prevents vendor lock-in and allows other engines within the Databricks platform to access the data directly.

Open Data Formats and Interoperability

By storing data in open formats, Lakebase ensures high interoperability within the Databricks ecosystem and beyond. This approach eliminates the need for complex and time-consuming ETL processes to move transactional data to the analytical layer, as the data is inherently accessible to both. This foundational integration streamlines data pipelines and provides a unified view of data across all workloads.

Key Technical Capabilities and Features

Databricks Lakebase offers a rich set of features that make it a compelling solution for modern data architectures:

PostgreSQL Compatibility: Lakebase provides full PostgreSQL semantics, including ACID transactions, indexing capabilities, and support for standard JDBC/psql clients. This familiarity allows developers to leverage existing skills and tools, minimizing the learning curve.
Fully Managed Service: Databricks handles the complexities of provisioning, scaling, patching, backups, and ensuring high availability, freeing up development teams to focus on application logic rather than database administration.
Managed Change Data Capture (CDC): A crucial feature, managed CDC ensures that operational data in Lakebase remains synchronized with Delta Lake tables for analytical consumption. This continuous synchronization is vital for keeping BI models and AI applications updated with the freshest transactional data.
Autoscaling (Lakebase Autoscaling): The latest iteration of Lakebase features intelligent autoscaling of compute resources. It dynamically adjusts Compute Units (CU) based on various metrics like CPU load, memory usage, and working set size, preventing performance bottlenecks and out-of-memory (OOM) issues. It also supports branching and instant restore, enhancing developer agility and operational resilience.
Databricks Apps Synergy: Lakebase is designed to serve as the transactional backend for Databricks Apps, enabling the creation and deployment of interactive applications directly on the platform, leveraging governed data and powerful analytics.

Governance, Security, and Cost Efficiency with Lakebase

Adopting Databricks Lakebase brings significant benefits in terms of data governance, security, and overall cost management, aligning with the principles of a modern data intelligence platform.

2.Reverse ETL with Lakebase simplifies data activation for operational analytics.

Unified Governance through Unity Catalog

One of Lakebase's most powerful integrations is with Unity Catalog, Databricks' unified governance solution. This integration provides a single pane of glass for managing data assets across the entire Databricks Data Intelligence Platform. Lakebase databases can be registered as catalogs within Unity Catalog, extending its robust governance framework to operational data. This means:

Consistent Access Control: Policies defined for your lakehouse data automatically apply to Lakebase, ensuring uniform security and access management across both operational and analytical workloads.
Centralized Auditing and Lineage: Unity Catalog provides comprehensive auditing capabilities and data lineage tracking for Lakebase assets, simplifying compliance and offering transparent insights into data flows.
Simplified Security Management: By unifying governance, organizations can reduce the complexity of managing security policies across disparate systems, enhancing overall data security posture.

Robust Security and Data Protection

Lakebase is designed with enterprise-grade security in mind, leveraging existing cloud infrastructure and Databricks' security features:

Network Integration: It integrates seamlessly with cloud networking services (e.g., Azure Private Link) for secure, private connectivity.
Identity Management: Integration with enterprise identity providers (e.g., Microsoft Entra ID) ensures secure authentication and authorization.
Data Encryption: Data is encrypted at rest and in transit, protecting sensitive information throughout its lifecycle.
High Availability and Disaster Recovery: As a fully managed service, Lakebase inherently provides features for high availability and point-in-time recovery, ensuring operational resilience.

Optimized Cost Efficiency

The architectural separation of compute and storage, coupled with advanced autoscaling capabilities, contributes to significant cost savings compared to traditional database architectures:

Pay-as-you-go Compute: With serverless and autoscaling compute, you only pay for the resources consumed during active processing, with the ability to scale down to zero when idle.
Low-Cost Storage: Leveraging economical cloud object storage for data persistence drastically reduces storage costs.
Reduced ETL Overhead: By eliminating the need for complex ETL pipelines between OLTP and OLAP, organizations save on infrastructure, development, and maintenance costs associated with data movement and transformation. This can lead to reported savings of 40-50% in many environments.

Lakebase in Action: Powering Real-Time Applications and AI Agents

Databricks Lakebase opens up new possibilities for building intelligent, data-driven applications that require both transactional capabilities and deep analytical insights. Its unified approach simplifies development and accelerates time-to-market for innovative solutions.

Real-World Use Cases

Personalized Recommendations: Build real-time recommendation engines that leverage fresh transactional data from Lakebase to provide immediate and highly relevant suggestions to users.
Customer Segmentation and Real-Time Updates: Maintain and update customer profiles and segments in real-time, enabling personalized experiences and targeted marketing campaigns.
Feature Stores for Machine Learning: Utilize Lakebase as a feature store to serve low-latency features to AI models, ensuring that predictions and decisions are based on the most current data.
Stateful AI Agents: Develop AI agents that can maintain conversational state and interact dynamically with users, using Lakebase as a reliable backend for transactional data.
Order Processing Systems: Implement operational applications that require high-frequency reads, writes, and updates, such as order management or inventory systems, directly on the Databricks platform.
Interactive Workflow Tools: Create interactive data applications and dashboards that allow users to both view analytical insights and perform transactional updates within the same environment.

A Practical Code Snippet

Developing with Lakebase feels familiar due to its PostgreSQL compatibility. Here’s a simple example demonstrating basic CRUD (Create, Read, Update, Delete) operations within a Lakebase table:

-- Create a schema for your application CREATE SCHEMA app AUTHORIZATION CURRENT_USER; -- Create a table to store session data for an AI agent CREATE TABLE app.sessions ( session_id UUID PRIMARY KEY, user_id TEXT NOT NULL, state JSONB NOT NULL, created_at TIMESTAMPTZ DEFAULT now(), updated_at TIMESTAMPTZ ); -- Create an index to optimize queries on agent status CREATE INDEX ON app.sessions ((state->>'agentStatus')); -- Insert a new session record INSERT INTO app.sessions(session_id, user_id, state) VALUES (gen_random_uuid(), 'u-123', '{"agentStatus":"active","score":0.82}'); -- Update an existing session's state UPDATE app.sessions SET state = jsonb_set(state, '{score}', '0.91'::jsonb), updated_at = now() WHERE user_id='u-123'; -- Query active sessions SELECT user_id, state->>'score' as current_score FROM app.sessions WHERE (state->>'agentStatus') = 'active';

This SQL snippet showcases how developers can interact with Lakebase using standard PostgreSQL syntax, enabling rapid application development within the Databricks environment.

The Lakebase Advantage: Performance and Reliability

Beyond its unified architecture, Lakebase is engineered for predictable performance and robust reliability, essential for mission-critical operational applications.

The radar chart above provides an opinionated comparison of Databricks Lakebase against traditional OLTP systems across several key attributes. Lakebase demonstrates superior performance predictability, dynamic scalability, cost efficiency, and ease of management, coupled with strong data governance due to its integration with Unity Catalog. Traditional OLTP systems, while effective for their specific purposes, often score lower in these cloud-native, unified data platform metrics.

Reliability Features for Business Continuity

Lakebase integrates several critical reliability features that ensure business continuity and data integrity:

Branching: This feature allows developers to create isolated, production-like environments for testing changes without affecting the main operational database. It promotes safer development practices and faster iteration cycles.
Instant Restore and Point-in-Time Recovery (PITR): In the event of data corruption or accidental deletion, Lakebase enables quick restoration to a previous state, minimizing downtime and ensuring data resilience.
High Availability: As a managed service, Lakebase is designed for high availability, with automated failover mechanisms and robust infrastructure ensuring continuous operation.

Validation and Troubleshooting: Ensuring a Smooth Lakebase Experience

Successful implementation and ongoing operation of Databricks Lakebase rely on proper validation and an understanding of common troubleshooting steps. This section provides a framework for ensuring your Lakebase deployment meets performance and reliability expectations.

An introductory video to Lakebase, explaining its core functionality and benefits for data apps and AI agents.

Key Validation Steps

After provisioning and configuring your Lakebase instance, it's crucial to perform a series of validation tests:

Connectivity Verification: Confirm successful connections from your applications or development tools (e.g., psql, JDBC clients) to the Lakebase instance. Ensure that Unity Catalog registration is visible and properly configured for governance.
Performance Baseline: Conduct baseline QPS (Queries Per Second) tests and monitor latency under expected load conditions. Validate that autoscaling events occur as anticipated and that performance targets are met.
Data Synchronization (CDC): Test the end-to-end data flow by inserting/updating records in Lakebase and verifying their timely appearance in Delta Lake tables via managed CDC. If reverse synchronization (Delta to Lakebase) is configured, validate that as well.
Governance and Security Checks: Confirm that Unity Catalog permissions are correctly enforced for Lakebase assets and that audit logs accurately reflect data access and modification events. Verify network security configurations (e.g., Private Link) are functioning as intended.

Common Troubleshooting Scenarios

While Lakebase is designed for stability, understanding potential issues and their resolutions is key to efficient operation:

Problem Area	Symptom	Potential Cause(s)	Troubleshooting Step(s)
Performance	High latency, slow queries, autoscaling not triggering as expected.	Inefficient queries, missing indexes, insufficient compute resources, working set exceeding memory.	Inspect query plans, add appropriate indexes, monitor CU utilization, review autoscaling logs, consider increasing initial compute capacity if persistently underperforming.
Data Sync (CDC)	Stale data in Delta Lake, sync job failures, data inconsistencies.	Incorrect Unity Catalog permissions, CDC configuration errors, network issues, regional feature limitations.	Verify Unity Catalog access for CDC process, check CDC job logs for errors, confirm network connectivity between Lakebase and Delta Lake, consult Databricks documentation for regional CDC availability.
Connectivity	Unable to connect from application, authentication failures.	Incorrect connection strings, firewall rules blocking access, misconfigured private endpoints, invalid credentials/tokens.	Double-check connection parameters, review network security group (NSG) and firewall rules, validate Private Link configuration, ensure correct user/service principal credentials.
Governance	Unauthorized access, unexpected data visibility, audit log discrepancies.	Incorrect Unity Catalog access policies, schema mismatches, misconfigured external locations.	Review and refine Unity Catalog grants on Lakebase catalogs and schemas, verify external location configurations, ensure consistent data object naming conventions.
Feature Limitations	Specific PostgreSQL features or extensions not working.	Managed environment restrictions, unsupported extensions.	Consult Databricks documentation for supported PostgreSQL versions and extensions in Lakebase. Adapt application logic to use supported alternatives if necessary.

By proactively monitoring and understanding these aspects, Cloud Solution Architects can ensure robust and efficient operation of Lakebase within their Databricks ecosystem.

Conclusion

Databricks Lakebase represents a pivotal advancement in data architecture, fundamentally reshaping how organizations approach operational and analytical workloads. By seamlessly integrating a fully managed PostgreSQL OLTP engine directly into the Databricks Data Intelligence Platform, Lakebase addresses the long-standing challenge of data fragmentation. This unification not only simplifies complex ETL processes and reduces operational overhead but also extends robust governance and security through Unity Catalog across the entire data estate. The innovative separation of compute and storage, coupled with intelligent autoscaling, delivers unparalleled cost efficiency and dynamic performance. For Cloud Solution Architects, Lakebase offers a compelling path to building scalable, real-time applications and sophisticated AI agents, leveraging fresh transactional data alongside comprehensive analytical insights—all within a single, consistent, and highly performant environment. This strategic evolution of the lakehouse architecture empowers enterprises to unlock new levels of agility, innovation, and data-driven decision-making.

Service Principals in Microsoft Power Platform

Shdsouza — Fri, 01 May 2026 17:18:09 GMT

Shared Passwords Are a Ticking Time Bomb

Let’s be blunt: if your Power Platform automation runs on a shared service account password that three people know, you are one resignation away from every flow going dark, one password reset, one MFA change… That is all it takes.

The fix? A service principal, a non‑human application identity in Microsoft Entra ID that authenticates with certificates, never logs in interactively, and keeps running no matter who leaves. In Power Platform, it shows up as an Application User in Dataverse and can own flows, manage Power BI datasets, run Dataverse operations, and power your CI/CD pipelines.

Unlike a traditional shared service account (which carries the risk of password expiration, MFA prompts breaking automation, and credentials being overshared), a service principal authenticates using certificates or client secrets, has no mailbox, no interactive login, and cannot be accidentally used by a person.

For organizations running Power Automate flows, calling APIs, running Power Platform Pipelines or managing environments programmatically, service principals offer a fundamentally more secure, auditable, and manageable identity. They enable least-privilege access, integrate cleanly with Conditional Access policies, and eliminate the single point of failure that comes with tying critical automation to an individual employee's account. This lack of tether to an individual account means workloads do not need to be reassigned once someone leaves your organization. Put more simply, if your Power Platform workloads are still running under a named user or a shared "svc_powerautomate@company.com" account, it's time to reconsider.

Microsoft Advisory: “Best practice is to use service principals as the preferred identity model for Power Automate wherever supported, because shared user‑based service accounts introduce security, audit, and operational risks.”

In this blog, we will highlight opportunities to strengthen your security posture across the Power Platform with Service Principals alongside ideas for when a Service Principal may not be applicable for your scenario.

Getting Started:

Setting up a service principal is straightforward. Everything is documented on Microsoft Learn:

🔗Register an app in Entra ID — Microsoft Learn

🔗Manage application users — Microsoft Learn

🔗Service principal owned flows — Microsoft Learn

There are also many good blogs about step-by-step setup of the service principal, assigning permissions and assigning to an application user we won't be covering the setup here.

Let's go into detail how you can utilize a Service Principal to secure your Power Platform workloads.

Where Service Principals Shine in the Power Platform

⚡ Power Automate

In Power Automate, service principals establish durable ownership and authentication for enterprise automation. Flows run under a non‑human application identity, eliminating dependency on individual users and preventing failures caused by password rotation, MFA enforcement, or user departure.

Flow Ownership:
Assign the service principal as the primary owner to ensure flows continue running regardless of personnel changes. Service Principal‑owned flows require either a Process license (~$150/flow/month, stackable up to 10 for 2.5M actions/day) or pay‑as‑you‑go billing via Azure.

Authentication:
Service principals authenticate using client credentials, removing the standard 90‑day connection expiry. Only the client secret or certificate has an expiration, which can be configured (up to 24 months). For production workloads, store secrets in Azure Key Vault with rotation alerts.

Dataverse Connector:
Dataverse is the only standard connector with native service principal sign‑in. Selecting Connect with Service Principal ensures all Dataverse actions execute under the application identity with full audit attribution.

Non‑Service Principal Connectors:
Connectors such as SharePoint, Outlook, and Teams require delegated user context by design. Where app‑only execution is required, the HTTP connector can be used to call Microsoft Graph with application permissions, introducing a premium dependency.

✓ Service Principal‑Friendly

✓ Microsoft Dataverse (native SP sign‑in)

✓ Custom connectors (app‑only OAuth)

✓ HTTP via Graph API (app permissions)

⚠ Requires User Context

⚠ SharePoint (standard connector)

⚠ Outlook / Office 365

⚠ Teams (many actions)

⚠ Planner, OneDrive

🔗Support for service principal owned flows - Power Automate | Microsoft Learn

🔗 Manage connections to Dataverse — Microsoft Learn

📱 Power Apps and Dataverse

Your Service Principal’s Application User executes Dataverse operations from flows triggered by model‑driven apps. Audit logs clearly separate automated changes from human activity. Assign custom security roles scoped to exact tables (skip System Admin), use separate Service Principalss per solution area, and know that canvas apps can trigger Service Principal‑backed flows behind the scenes.

🔗Create a Dataverse application user — Microsoft Learn

📊 Power BI

This is where Service Principals quietly save organizations from one of the most common Power BI failures: dataset refresh breaking because the owner left. Take over semantic model ownership via the REST API and refreshes never fail from expired tokens again.

Workspace access: add Service Principal as Member or Admin
Semantic model ownership takeover via REST API
Automated refresh from PowerShell, Logic Apps, Azure Data Factory, or custom apps
XMLA endpoint access for model deployment and DAX queries (Premium/Fabric)
App Owns Data embedding for external users without Power BI licenses

Requirement: Tenant admin enables “Service principals can use Fabric APIs”. Service Principal added to workspace. API refresh needs Premium, Premium Per User, or Fabric capacity.

🔗Automate Premium tasks with Service Principals — Microsoft Learn

🔗Enhanced refresh REST API — Microsoft Learn

🛠️ Application Lifecycle Management (ALM) with Pipelines

Power Platform Pipelines are a built-in option for application lifecycle management native to the Power Platform. Pipelines bring ALM automation and continuous integration and continuous delivery (CI/CD) functionality into a native service that's designed to be usable by all members of your low-code team, regardless of technical capability. To learn how to set up pipelines in Power Platform, Microsoft has some fantastic documentation as well as a learning path on Learn.microsoft to guide you through the process.

We can utilize Service Principals in Pipelines as well. By default, a pipeline deploys as the requesting maker, meaning the maker needs elevated permissions to deploy to the target environment. Delegated deployments with Service Principals allow deployment without the maker needing elevated permissions in the downstream environment. The pipeline can run as a service principal (or pipeline stage owner), allowing makers to deploy without needing elevated permissions in target environments like production. Approvals may be required for security reasons, and can be automated or manual depending on your security requirements.

🔗PAC CLI: createserviceprincipal - Microsoft Learn

🛡️ Alternatives for Secure Workloads

While service principals offer an avenue to securing most workload scenarios on the Power Platform, they cannot cover all workload scenarios. In these cases, there are alternative approaches.

Azure Logic Apps support system-assigned and user-assigned managed identities, allowing workflows to authenticate directly to Azure resources without storing or managing any login credentials. This can offer an option when your workload can live outside Power Automate.

Some teams opt to retain shared service accounts. This is not best practice, but when undertaken, there are steps to harden these accounts. Dedicated Entra Conditional Access policies to enforce MFA, restricting sign in locations and devices can provide additional protection. For password rotation, Azure Key Vault centralizes credential storage and enables automatic secret rotation, reducing the risk of stale or exposed passwords.

Each of these approaches can be layered or combined with service principals depending on your organization's risk tolerance, licensing constraints, and connector support limitations.

🔗Governing Entra service accounts — Microsoft Learn

All Your Options at a Glance

Not sure which approach fits? Here is every option compared — from the gold standard to the fallback you hope to leave behind:

Option	MS Rec?	Key Benefit	Main Tradeoff	Security
SP + Process License	Yes	Gold standard. Certificate auth, clean audit.	~$150/flow/mo. Solutions required.	Highest
SP + Flow Groups	Yes (GA May ’26)	1 license across up to 25 flows.	Shared 250K/day action pool.	Highest
SP + Pay-as-you-go	Yes	Available today. No upfront cost.	Variable cost. Azure sub needed.	Highest
Hybrid: SP + SA	Pragmatic	SP owns; SA for delegated connectors.	SA still exists for connections.	Med‑High
Hardened SA Only	No	Simplest and no migration needed.	Interactive login risk. MS discourages.	Low
Logic Apps + MI	Yes (diff product)	Zero credentials. Consumption billing.	Full platform migration required.	Highest

⚠ All costs are approximate and vary by enterprise agreement. Verify with your Microsoft representative.

Microsoft‑validated: The dedicated service account fallback is a supported approach, not a workaround. Any alternative to service principals would need to be designed and accepted at the customer’s own risk.

Summary

Service principals replace shared, user‑based accounts with a non‑human application identity that is secure, auditable, and resilient. Eliminating password risk, improving governance, and ensuring Power Platform workloads continue running independent of employee lifecycle events

across the entire platform. The result is higher security, operational stability, and governance consistency.

The Bottom Line

Service principals are not optional polish. They are how Microsoft expects you to run enterprise automation in 2026. One Entra ID app registration. Four products. Zero shared passwords.

✓ Blocking interactive admin logins is correct. No supported scenario requires it.

✓ Hybrid model when needed. Service Principals where supported, hardened service accounts where not. Both documented by Microsoft.

✓ Start with pay‑as‑you‑go. Pilot a batch. Measure costs. Then commit. No upfront purchase needed.

Authors: Sheldon Dsouza + Marc Lotorto | Contributors: Rasha Al-Silmi, Ahmed Shaalan, Josh Flicker

All guidance aligned with Microsoft Learn and validated by Microsoft Advisory, April 2026

The Journey of Copilot: From Setup to Mastery for Azure SAP customers

AnuradhaKarnam — Fri, 01 May 2026 17:17:14 GMT

Introduction:

GitHub Copilot integrates as an extension or plugin within developer tools commonly used in SAP and Azure scenarios, such as Visual Studio, Visual Studio Code, and other supported IDEs. These tools are often used alongside SAP development (e.g., ABAP, CAP, or integrations with S/4HANA and Azure services).

Before you begin, ensure you have access to Copilot, through an organizational license (common in enterprise environments).

Install GitHub Copilot

Step 1: Install Required Extensions

Open Visual Studio Code
Go to Extensions (Ctrl + Shift + X)
Install the following extensions:

GitHub Copilot
GitHub Copilot Chat
GitHub Copilot for Azure (Microsoft extension)

When installing the Azure extension, it may prompt you to install additional Azure tools, accepting all required components.

Step 2: Sign in and Authenticate

Sign in to your GitHub account
Sign in to your Azure account
Complete authentication in the browser
Return to VS Code

Both logins are required:

GitHub → enables Copilot
Azure → enables Azure resource access and tools

Step 3: Enable and Verify Setup

Open Copilot Chat (Ctrl + Alt + I)
Check that Copilot is active
Verify Azure integration by typing a test prompt:
What Azure resources are deployed and running in my subscription?

If you get a response → setup is successful

Step 4: Configure Azure Context (Important for SAP)

Set your Azure tenant / subscription (Entra ID)
Ensure correct environment for:

SAP on Azure (S/4HANA, SAP NetWeaver)
SAP BTP extensions

Optional: Enable Agent Mode for automation tasks (deployments, scripts)

Get Started in Your SAP Development Environment

Open your preferred IDE (Visual Studio, VS Code, or Eclipse with SAP tooling)
Access the Copilot chat or assistant panel within the IDE
Sign in with your GitHub account (and organizational account if required)
Start using Copilot in your SAP development scenarios

Use Copilot for SAP Workloads

Inline suggestions

Get real-time code suggestions for SAP-related languages (e.g., JavaScript, Java, ABAP extensions, CAP models)

Ask questions in chat

Understand existing logic, SAP APIs, or integration patterns (e.g., “Explain this service” or “How does this SAP function work?”)

Generate and improve code

Create boilerplate logic, unit tests, and integration code faster
Identify performance or design improvements in existing SAP code

Enhance with SAP Context

Provide additional context (files, APIs, or SAP objects) to improve suggestions
Optionally connect Copilot to SAP data or services using enterprise integrations
Use Copilot to support:

SAP BTP extensions
S/4HANA integrations
Fiori/UI5 development and APIs

Once you start interacting with Copilot, it acts as an AI assistant within your SAP development workflow, helping you write code faster, understand existing logic, and accelerate innovation across your SAP and Azure landscape.

The Hidden Layer: Network Configuration for SAP Customers

As you begin using GitHub Copilot within your SAP development and integration environment, you may notice performance differences, especially when working within corporate networks. In most cases, Copilot connects securely to GitHub services over the internet using HTTPS, without requiring additional setup. However, in SAP enterprise environments where strict governance, security policies, and compliance controls are in place, network traffic is often routed through proxies, firewalls, or VPNs.

What You Need to Know

Copilot may require additional configuration when operating behind corporate proxies or firewalls
Proxy settings can be configured:

Directly within your IDE
Or through environment variables such as HTTP_PROXY and HTTPS_PROXY

Authentication to enterprise proxies may require:

Basic credentials
Or enterprise mechanisms such as Kerberos-based authentication

Enterprise Considerations for SAP Landscapes

Organizations may require custom SSL certificates for secure outbound connections
Network security policies may restrict access to external services
Required Copilot and GitHub endpoints must be allowed to ensure connectivity

Why This Matters for SAP Customers

In SAP environments especially those involving S/4HANA, SAP BTP, or hybrid/on‑premise systems network security is tightly controlled. Proper configuration ensures that Copilot can securely interact with external services while still complying with enterprise security standards. Once configured correctly, Copilot integrates seamlessly into your SAP development workflow, enabling secure, reliable, and high‑performance AI-assisted development within your governed enterprise environment.

Configure Network Settings (if required) for Azure SAP Environments

In Azure‑hosted SAP landscapes (such as S/4HANA on Azure, SAP BTP, or hybrid environments), network configuration plays a critical role in enabling GitHub Copilot securely. Network setup is primarily required in enterprise environments where security controls such as proxies, firewalls, VPNs, or Azure networking policies are enforced.

Default Behavior

GitHub Copilot connects securely over HTTPS
No additional configuration is required in open network environments

Proxy & Enterprise Network Configuration

If your Azure SAP environment uses controlled outbound access:

Configure proxy settings:
- HTTP_PROXY
- HTTPS_PROXY

Directly within your IDE (Visual Studio, VS Code)
Or via environment variables:

Supported authentication methods:

Basic authentication
Kerberos (common in enterprise identity setups)

Additional considerations:

Ensure required GitHub/Copilot endpoints are allowed in Azure firewall or network security groups
Install custom SSL certificates if your organization uses SSL inspection

Note: Visual Studio typically inherits Windows/Azure VM proxy settings

Troubleshooting Network Issues in Azure SAP Scenarios

If Copilot stops responding or behaves inconsistently, the issue is often related to enterprise network controls in Azure or hybrid SAP architectures.

Common Causes

Proxy or firewall blocking outbound connectivity
SSL certificate validation failures
VPN or private network restrictions (ExpressRoute / private endpoints)

Quick Diagnostics: Test connectivity from your Azure VM or development machine:

curl --verbose https://copilot-proxy.githubusercontent.com/_ping

If using a proxy: curl --verbose -x http://PROXY:PORT -i -L https://copilot-proxy.githubusercontent.com/_ping

HTTP 200 → Connectivity is working

Errors → Network blocking or configuration issue

Recommended Troubleshooting Steps

Verify proxy settings are correctly configured
Check SSL certificates and trust chain
Review Azure firewall, NSG, or proxy rules
Validate required endpoints are reachable
Enable verbose logs or diagnostics in your IDE for deeper analysis

Best Practice for Azure SAP Customers

Adopt a structured troubleshooting approach:

Validate connectivity
Trace the network path (proxy, firewall, DNS)
Fix configuration issues systematically

This aligns with the governance and operational discipline already used in SAP and Azure environments.

Outcome: A Confident Copilot User in Azure SAP

By following this approach, you move beyond basic usage and gain full control of Copilot within your enterprise landscape.

You will be able to:

Deploy and use Copilot across Azure SAP environments
Integrate it securely within enterprise networking constraints
Troubleshoot issues with confidence using systematic diagnostics

Conclusion:

GitHub Copilot is no longer a black box, it becomes a trusted, secure, and intelligent AI assistant seamlessly integrated into your Azure and SAP development ecosystem. As you adopt it into your workflow, development becomes faster, cleaner, and more efficient. More importantly, you gain a reliable partner that enhances productivity and supports innovation, ensuring that you are no longer coding alone, but collaborating with AI to deliver better outcomes.

Reference links:

https://docs.github.com/en/copilot/how-tos/set-up/install-copilot-extension

Get Started with GitHub Copilot - Visual Studio (Windows) | Microsoft Learn

https://learn.microsoft.com/en-us/azure/developer/github-copilot-azure/get-started?pivots=visual-studio-code

Network settings for GitHub Copilot - GitHub Docs

Troubleshooting network errors for GitHub Copilot - GitHub Docs

GitHub Copilot for SAP ABAP in VS Code: Setup Guid... - SAP Community

VDI, Teams, and what’s changing in 2026: VBSS becomes VMSS, and eCDN lands in the core license

jchristie — Thu, 30 Apr 2026 07:50:02 GMT

Audience: Mission Critical customers running Microsoft Teams on virtualized desktop platforms (Citrix, AVD, Windows 365, VMware/Omnissa Horizon).

TL;DR: Two Teams-on-VDI changes are converging: VMSS is already in Public Preview today as the successor to VBSS in the new VDI solution for Teams (Microsoft Learn - Screen sharing), and Microsoft eCDN is now included in Teams core license. This post previews the guidance our Support for Mission Critical (SfMC) Cloud Solution Architects (CSAs) are already walking customers through - because the cost of finding these issues in production is always higher than finding them in a pilot.

Why we’re flagging this now

SfMC exists to get ahead of changes like these. The SfMC CSA role is built on a simple principle: be a trusted advisor embedded alongside the customer team, not a reactive support line. SfMC CSAs work hand-in-hand with platform, network, security and service-ownership teams to build a deep “know-me” picture of the customer - their gold-image strategy, their VDI vendors, their peering topology, their CAB cadence, the history of what was tried and what didn’t stick. That context is the reason a readiness review lands in weeks, not months: your SfMC CSA isn’t starting from zero, they’re starting from knowing the estate.

Goodbye VBSS, hello VMSS - and it’s here now

Teams on VDI has used Video Based Screen Sharing (VBSS) for years - an efficient, encoded video stream for screen shares. That approach is being replaced by Virtual Machine Screen Sharing (VMSS) as part of Microsoft’s New VDI solution for Teams.

This isn’t a future roadmap item - VMSS is available in Public Preview today across Azure Virtual Desktop, Windows 365, Citrix and Amazon WorkSpaces, with Omnissa following. Microsoft’s guidance and support matrix is live on Microsoft Learn: New VDI solution for Teams - Screen sharing. If you have users on a pilot ring on VDI, you can light this up now, simply by activating Public Preview for them.

Spot the screen sharing stream no longer being offloaded to client side slimcore

Support depends on three things moving together: the Teams client on the session host, the virtualization vendor’s optimization component (Citrix HDX / AVD Multimedia Redirection / VMware-Omnissa Media Optimization), and the endpoint client (Windows App, Citrix Workspace App, Horizon Client). Where any one of those lags, screen share quietly falls back to a lesser modality - users don’t raise tickets, they just tolerate worse quality.

Because VMSS is already in preview, there’s a real window to get this right before it becomes the default path. On Mission Critical engagements, SfMC CSAs are already sitting with customer teams on VMSS readiness reviews: confirming client and plugin versions across the gold-image estate, rebuilding CQD dashboards so the baseline survives the cutover, and flagging any inline network appliance that still assumes the old VBSS flow. The “know-me” picture the SfMC CSA has built up makes that work fast - they already know which plugin versions the desktop team is running and which CAB window the next image refresh lands in.

Microsoft eCDN is now in the core Teams license

Microsoft eCDN - previously a paid add-on - is now included in the Microsoft Teams core license. It’s a WebRTC-based peer-to-peer mesh that offloads large-scale town halls and live events from the corporate WAN by peering video between clients on the same site.

If the business case for the add-on never cleared, that objection is gone. But “included” doesn’t mean “working”. The failure mode we see is consistent: customers enable eCDN because “it’s free now”, but the peering never works - because the client-to-client path is blocked by security controls nobody remembers adding. The town hall runs, the WAN still saturates, the CIO asks why the thing that was supposed to fix it didn’t.

Example eCDN portal dashboard

The VDI infrastructure question

Both changes elevate something that has always mattered but rarely been tested: VDI-to-VDI network reachability. The new Teams client needs to talk to Microsoft 365 media endpoints (usually already open) and to other VDI instances on the same site for eCDN peering.

That second requirement is where customers are consistently caught out. Most VDI builds treat each session host as an island - east-west traffic between session hosts is blocked by NSG, hypervisor firewall, or micro-segmentation policy, because it was never needed. With eCDN in the box, it is now needed - and the blocks are often in places the virtualization team doesn’t own.

This is where working hand-in-hand with the customer team pays off. The SfMC CSA convenes the platform, network, and security owners, translates the platform change into each team’s language, and makes sure nothing falls through the gaps between them. The specific hostnames, IP ranges, UDP/TCP port requirements, and peering-group configuration are all on Microsoft Learn (links below) - the hard work is operationalizing them against your estate, and that’s the work your SfMC CSA is built to drive.

If two or more of these apply to your estate, book the conversation with your SfMC CSA now:

Client version sprawl - multiple Teams versions in flight across gold images, or a long tail of unpatched Citrix Workspace App / Windows App / Horizon Client.

Missing or partial CQD data - gaps in building/subnet mapping, “unknown” network location for a meaningful share of streams, dashboards still filtered on legacy VBSS modality tags.

Recent east-west firewall changes - new micro-segmentation rollout, zero-trust project, or NSG rule consolidation in the last 12 months.

Recent live-event pain - WAN saturation, buffering, or join failures on the last town hall.

No eCDN subnet map, or a map that predates your current site/subnet topology.

Proxy or TLS-inspection changes forcing Teams media through an inspection device rather than bypassing it.

VPN full-tunnel without eCDN VPN exclusion.

Upcoming large broadcast in the next 90 days.

Closing thought

VMSS is in Public Preview today and eCDN is already in your Teams license. The window to pilot, validate and harden is open right now - and it closes the moment either of these becomes the default path for your users.

That’s what Support for Mission Critical is built for: Cloud Solution Architects working shoulder-to-shoulder with your team as trusted advisors, investing the time to genuinely know your estate - your platforms, your people, your change windows, your risks - so that when a shift like VMSS or eCDN arrives, the remediation plan is already half-written. Not a ticket-shop. A partnership.

If you’re running Teams on VDI at scale and you haven’t had the VMSS + eCDN conversation with your SfMC CSA yet - that’s the next call to book.

References

New VDI solution for Teams - Screen sharing (VMSS, Public Preview) - https://learn.microsoft.com/en-us/microsoftteams/vdi-2#screen-sharing

New VDI solution for Teams (overview) - https://learn.microsoft.com/en-us/microsoftteams/vdi-2

Microsoft Teams for VDI - install requirements - https://learn.microsoft.com/en-us/microsoftteams/teams-client-vdi-requirements-deploy

Microsoft eCDN networking requirements - https://learn.microsoft.com/en-us/ecdn/technical-documentation/network-requirements

eCDN peering groups and restrictions - https://learn.microsoft.com/en-us/ecdn/how-to/set-up-peering-groups

Microsoft 365 URLs and IP address ranges - https://learn.microsoft.com/en-us/microsoft-365/enterprise/urls-and-ip-address-ranges

Enterprise Security Assessment: A Strategic Lens for Mission Critical Environments

lakprasad — Wed, 29 Apr 2026 21:34:39 GMT

Understanding Enterprise Security at Scale

Understanding security posture at scale requires more than isolated control reviews or point‑in‑time assessments. The Enterprise Security Assessment (ESA) helps organizations understand their security posture across Azure, Microsoft 365, and hybrid environments from a true enterprise perspective. Instead of assessing individual services or workloads in isolation, ESA provides a single, enterprise‑wide view of security.

By examining identity, data security, endpoints, threat protection, and cloud infrastructure together, ESA helps uncover gaps that often span multiple teams and platforms. This broader perspective enables clearer prioritization, stronger alignment across security teams, and a more resilient foundation for long‑term security improvement.

ESA complements other Microsoft assessments, such as workload‑specific reviews, by connecting the bigger picture - to align security priorities across teams and platforms, fostering a more cohesive and resilient security approach.

From Standard Engagement to Strategic Partnership

An Enterprise Security Assessment is typically delivered as a focused engagement designed to establish an enterprise‑wide view of security posture. At Microsoft, we begin by reviewing Secure Score insights, analyzing a defined set of core security datasets, and correlating those signals across Azure and Microsoft 365.

For many organizations, this approach works well. Collecting and evaluating these datasets provides a high‑level understanding of security posture, highlights common gaps, and identifies priority improvement areas. In standard enterprise environments, ESA delivers actionable insights with minimal disruption and sets a solid foundation for security improvements.

How ESA Evolves in Mission‑Critical Environments

In large or mission‑critical environments, security is often distributed across multiple teams and tools. Operational constraints, regulatory requirements, and business dependencies introduce complexity that standard assessments cannot fully capture.

For mission‑critical customers, ESA goes beyond a baseline review and becomes more consultative. This typically includes:

📝 Structured discovery sessions across multiple security domains

🤝 Deep‑dive workshops with specialized teams

🎯 Validation of findings against real‑world operating models

🔄 Iterative analysis to validate findings against real operational conditions

This ensures recommendations reflect how security is actually managed, not just how it is documented.

Why Going Deeper Matters to Customers

For organizations operating at scale, this consultative ESA approach delivers significantly more than a standard readout:

A realistic, enterprise‑wide understanding of security posture, grounded in actual configurations and operating models
Clear visibility into cross‑team dependencies and systemic risks
Prioritized recommendations aligned to existing licenses, third‑party tools, and regulatory requirements
A realistic, phased security roadmap focused on adoption, not theory

The result is a clear starting point for security improvements that teams can execute with confidence.

A Continuous Improvement Model

ESA is not a one‑time exercise. For most customers, it becomes the foundation for ongoing security maturity.

Once a baseline is established, future ESAs are faster and more efficient, allowing organizations to track progress, validate improvements, and maintain alignment as environments evolve. Over time, ESA functions as an annual enterprise security health check, supported by follow‑up reviews and continuous improvement.

In mission‑critical environments, this means:

The first ESA requires deeper engagement investment
Building cross-team alignment takes time
Future assessments become smoother and more efficient once a baseline is established

Over time, ESA functions as an enterprise security health check that supports continuous improvement. It works best when treated as a starting point for continuous improvement, and Enterprise Security Alignment.

What Customers Gain from an Enterprise Security Assessment

A true enterprise view
Visibility across identity, data, devices, cloud workloads, and threat signals - without losing sight of critical details.
A customized security roadmap
Recommendations aligned to existing licenses, third‑party tools, hybrid footprints, and regulatory requirements - making adoption realistic, not aspirational.
Momentum and measurability
Many organizations track progress using dashboards or scorecards to measure improvement and sustain focus over time.
Repeatability
Once a baseline is established, future ESAs become easier and more efficient - serving as a regular health check rather than a brand‑new effort.
A consultative model
ESA delivers far more value than a one‑time assessment by fostering collaboration, shared understanding, and long‑term alignment.

A Foundation for Continuous Improvement

Enterprise security is complex, especially at scale. In mission‑critical environments, security success depends on embracing complexity, aligning teams, and moving beyond a standard assessment playbook.

An Enterprise Security Assessment is more than a snapshot. It’s an opportunity to build alignment, inform strategy, and create a resilient security foundation that evolves with the organization.

SharePoint Automatic Version History Cleanup (Intelligent Versioning)

jeresmith — Sat, 09 May 2026 16:53:42 GMT

What is SharePoint Automatic Version History Cleanup?

SharePoint Automatic Version History Cleanup is a feature in Microsoft 365 (SharePoint Online and OneDrive) that automatically manages and prunes file version history based on the age of versions and file activity. It is part of the “Version History Limits” functionality that gives admins control over how many versions to keep and for how long. When this Automatic mode is enabled (often referred to as Intelligent Versioning), SharePoint will no longer retain every single version up to the static limit indiscriminately. Instead, it will “thin out” older versions over time, keeping a higher density of recent versions and progressively fewer versions as they age. This ensures that most day-to-day edits remain recoverable, while redundant or stale versions from long ago are cleaned up.

Crucially, automatic cleanup does not require administrators or users to manually delete versions or set specific limits for each library. In the traditional model (Manual versioning), admins or site owners had to configure each library to keep a fixed number of versions (with a minimum of 100) and possibly specify a time-based deletion for older versions. In contrast, the Automatic setting uses built-in logic to manage versions dynamically. Microsoft’s internal testing and customer feedback guided this feature to address the major pain point of runaway version storage while maintaining “strong recoverability” for files.

Key characteristics of Automatic (Intelligent) Versioning:

Time-based retention algorithm: It looks at the age of each version and the file’s edit frequency to decide which versions to keep. Recent changes are kept in detail, whereas older changes are pruned, keeping only periodic snapshots.

Dynamic, ongoing cleanup: As new versions are created, older ones are evaluated and trimmed automatically in the background. This is not a one-time job, but a continuous policy is applied to the library.

Wider recovery window with fewer versions: Users still have access to versions spanning a long time period (e.g. many months or years), but without the full count of every minor change. The system preserves important restore points (like the first version of each week or day), assuming those are more valuable for recovery than every tiny edit.

Storage space optimization: By cutting down on redundant older versions, organizations see dramatic storage savings. Microsoft reports up to a 96% reduction in version storage over a 6-month period using automatic trimming, compared to keeping all versions under a 500-count limit.

Still protective of current versions: The most recent versions (within the last days or weeks) are generally all retained. The current file version is never deleted by the system, and recent version history remains robust for auditing and quick rollback needs.

Applies to Office documents (and more): Intelligent versioning is particularly beneficial for Office files (Word, Excel, PowerPoint) that save frequently, but it works for any files in SharePoint/OneDrive.

How the Automatic Cleanup Algorithm Works

When Automatic version limit is in effect, SharePoint uses a built-in tiered retention algorithm based on version age. In simple terms, the older a version is, the less frequently it’s kept. Here is a summary of the default intelligent retention logic:

Age of File Version	Retention by Automatic Cleanup
0–30 days old	Keep all versions. Every saved version from the last 30 days is preserved (upto 500 versions). This ensures you can track all recent changes in detail.
31–60 days old	Keep hourly versions. For versions in this range, the system prunes away some duplicates, aiming to retain roughly one version per hour of edit activity. In practice, if multiple versions were saved within the same hour, only the latest from that hour might be kept.
61–180 days old (2–6 mo.)	Keep daily versions. Versions older than two months get further thinned out to about one per day, preserving a daily snapshot of the file’s state.
Over 180 days old	Keep weekly versions. Very old versions (beyond ~6 months) are trimmed to approximately one per week, maintaining a weekly snapshot over long periods.

This tiered approach means that if a file is actively edited, you’ll have all of its versions from the past month, then a representative sampling of versions as you go back in time (hourly→daily→weekly). In effect, the algorithm removes redundant intermediate saves that are likely low-value (e.g. dozens of near-identical saves due to auto-save in a short period) while still keeping a timeline of the document’s evolution. If a file hasn’t been edited in a long time, its last saved versions will remain available at least until they hit the weekly or daily thresholds.

Maximum Number of Versions: Even under Automatic mode, SharePoint will not keep more than 500 versions of a file. This is a hard cap that remains in place for now. If a file continues to be edited very heavily over months or years, hitting 500 versions, the oldest versions will be trimmed to honor the cap. In practice, however, most files are unlikely to hit 500 retained versions under the automatic algorithm, because many interim versions would already be pruned by age. The 500 limit mainly serves as a safety net.

Expiration Labels in Version History UI: Once you switch a library or site to Automatic limits, you may notice in the Version History view that older versions get an “expiration date” label. These dates indicate when a given version is scheduled to be removed by the algorithm. For example, a version might show “Expires on 5/10/2026”, meaning the system will automatically delete it on that date (unless it gets preserved longer due to other rules). The most recent version is never assigned an expiration date (it does not expire at all), and very new versions may show “Never expires” until they age beyond the no-trim window.

Example of Automatic Cleanup in Action

Imagine a project plan (Excel file) that multiple team members edit daily over the course of the year. Under the old policy (500 versions, no expiration), if the team saves changes frequently, they might hit 500 versions in a few months, after which SharePoint starts dropping the oldest versions on each new save. If the editing is less frequent, they might not hit 500 for a long time, but all versions (even trivial ones) from throughout the year remain, eating storage.

With Automatic version cleanup enabled, SharePoint will keep every version for the first 30 days of rapid collaboration, then automatically trim and compress the version history:

After a few months, you’ll still have complete daily snapshots of how the file looked each day, but not every single save from, say, 4 months ago.

After a year, you might have weekly snapshots remaining from the early months.

The team can restore the file to any week in the past year, or any day in the past 6 months, or any hour in the past 60 days, etc., giving ample recovery points.

The storage used by this file’s version history will be dramatically lower than it would be under the old scheme (potentially just a few dozen versions retained instead of hundreds). In Microsoft’s example, automatic trimming yields ~96% storage reduction for versions over six months.

From the user perspective, nothing special needs to be done — version cleanup happens behind the scenes. Users still go to Version History on a document and see a list of versions, but with fewer ultra-fine-grained ones as they get older. Admins benefit by not having to constantly monitor or manually delete old versions to free space.

Configuring Automatic Version History Cleanup in SharePoint Online

Setting up Automatic version cleanup requires adjusting your SharePoint Online versioning settings at the appropriate level. Here’s how to configure it:

Organization-Level Default Setting (SharePoint Admin Center)

To enable intelligent version management across your tenant, navigate to your SharePoint admin center Settings and Version history limits.

Once this is saved, SharePoint Online will use Automatic (intelligent) version limits by default on any new libraries created in your tenant. Existing sites and libraries, however, do not retroactively change just by toggling this setting. They will continue with their current versioning settings until you update them (see below).

Verifying the setting: It may take some time for the new setting to propagate. You can confirm that it’s in effect by creating a new document library on a site (after enabling Automatic) and checking the library’s version settings or testing with a file.

If for any reason you need to switch back to manual settings globally, you can do so similarly in the Admin Center by choosing the manual option and specifying the number of versions and expiration days (if any). By default that might revert to 500 versions, no expiration. You can also manage this via PowerShell (see next section).

Site-Level and Library-Level Configuration

There are scenarios where you might not want to use the organization’s default for every site or library. SharePoint allows breaking the inheritance:

Site-level limits: A SharePoint site (site collection) can have its own version history policy that overrides the tenant by default for all libraries in that site. However, as of now, Microsoft’s UI does not provide a direct way to set site-level versioning in the admin center. You must use PowerShell cmdlets to configure a site’s setting. For example, to enable Automatic mode on a specific site (if the tenant default is not already automatic), you would run:

This flags that site to use automatic version limits for new libraries. (Add the -ApplyToExistingDocumentLibraries switch if you want to apply it to all current libraries on that site as well. Otherwise, existing libraries remain as they were, and only newly created libraries on that site use the new policy.)

Library-level limits: Site owners or admins can configure individual document libraries to have their own version limit settings, overriding both site and org defaults for that library. This is done either through the Library Settings in the SharePoint site UI or via PowerShell. In the library’s settings page (under “Versioning settings”), modern SharePoint should expose fields for the version limit and expiration if the admin has allowed that. For example, you might set one specific library to manual 100 versions, while the rest of the site follows Automatic, or vice versa, depending on needs.

In PowerShell, you can use Set-SPOListVersionPolicy to manage a specific library’s policy. For instance, to turn on Automatic for one library:

Or to set a manual limit on a library (say 200 versions, no expiration):

You can also specify a time limit (ExpireVersionsAfterDays) in combination with the version count if needed.

Keep in mind that lowering version limits on an existing library does not instantly delete all the extra versions above the new threshold. Instead, SharePoint will trim them gradually as new versions are added, to avoid large sudden deletions. According to Microsoft, if you reduce a library’s limit from 500 to 300, the next time someone edits a file that has, say, 500 versions, the system will purge up to 20 of the oldest versions on that save, then another 20 on the next save, and so on until the file complies with the 300 limit. This process prevents performance issues from mass deletion. (If you want immediate cleanup of a huge backlog of versions, consider using the trim job approach below.)

Using PowerShell for Tenant-Level Settings

For completeness, note that you can enable or disable the automatic versioning feature across the tenant via PowerShell as well. The relevant property is EnableAutoExpirationVersionTrim on the tenant:

To enable Automatic globally (equivalent to selecting Automatic in Admin Center):

This turns on the new “intelligent” version limits at the org level. After running this, you would typically also specify what you want the manual limits to be, in case you switch back or for any site still using manual. By default when turning on auto, SharePoint sets the global MajorVersionLimit to 500 and ExpireVersionsAfterDays to 0 (no time limit) behind the scenes.

To disable Automatic and revert to manual, you might run:

(This example sets a manual policy of 500 versions, no expiration. Adjust the numbers as needed, and note the UI minimums of 100 versions / 30 days if setting via UI.)

There are also PowerShell cmdlets to apply settings in bulk to sites. For example, you can iterate through all site collections and activate intelligent versioning for each one using a loop with Set-SPOSite -EnableAutoExpirationVersionTrim $true, as demonstrated in the SharePoint Diary blog. Use caution with such scripts, and run them in batches or during off-hours if you have many sites.

Trimming Existing Version History (On-Demand Cleanup Jobs)

Enabling Automatic mode will govern the retention of new versions going forward. But what about old versions that already exist from before you changed the setting? Those will not magically disappear the moment you switch modes. For example, if a library had 400 versions of a file and you turned on auto (or lowered the manual limit to 100), those 400 will still be there until new edits trigger the algorithm to clean up gradually. In some cases you might want to immediately reclaim storage by clearing out old versions in bulk, according to the new policy or other criteria. This is where SharePoint’s Version Trimming Jobs come in.

On-demand trimming allows admins to explicitly remove versions from existing files in a site or library. Microsoft provides PowerShell cmdlets to queue these jobs, which run asynchronously on the server to delete versions matching certain filters. There are three types of trim operations you can choose from:

Manual expiration trim: Delete versions older than a specified date threshold (e.g., remove all versions older than 180 days).

Manual count-based trim: Delete the oldest versions exceeding a specified count (e.g., keep the latest 100 versions and remove the rest).

Automatic trim: Apply the same intelligent algorithm to existing versions. This will simulate what the Automatic mode would have done and remove the excess versions accordingly (older ones may be outright deleted or assigned expiration dates depending on their age).

To use these, you’d run commands like:

1 # Example: Trim versions older than 180 days on an entire site

2 New-SPOSiteFileVersionBatchDeleteJob -Identity https://<siteURL> -DeleteBeforeDays 180

3 # Example: Trim to a count limit of 100 on a specific doc library

4 New-SPOListFileVersionBatchDeleteJob -Site https://<siteURL> -List "<LibraryName>" -MajorVersionLimit 100

5 # Example: Apply the automatic algorithm to trim versions on a site

6 New-SPOSiteFileVersionBatchDeleteJob -Identity https://<siteURL> -Automatic

These jobs permanently delete the matching versions (bypassing the recycle bin, so they cannot be recovered once trimmed). Microsoft therefore strongly recommends running a “What-if” analysis first: you can generate a Version Storage Report for a site or library and then simulate the trim to see how many versions would be deleted and how much space saved. This helps validate that you won’t accidentally remove something critical. The “What-if” process involves an auditing cmdlet (New-SPOSiteFileVersionExpirationReportJob) that produces a CSV of versions and their would-be deletion status under given rules, which you can review.

Trimming jobs run in the background and can take a significant amount of time for large libraries (possibly hours or days), particularly if thousands of versions are being evaluated. They tend to run during off-peak hours automatically. You can check the status of a job via PowerShell or the SharePoint admin center (there’s a page listing version trim jobs and their progress).

Important: Always inform site owners before trimming versions, and ideally take a backup or export of version history if the content is mission-critical. Once a version is deleted by a trim job, it’s gone for good (unless you restore the entire site from a backup). Trimming is irreversible and bypasses the recycle bin1.

Best Practices for Managing Version History in SharePoint Online

For IT administrators and power users managing SharePoint, here are best practices and considerations to get the most out of version history while avoiding pitfalls:

Adopt Automatic Versioning for Most Scenarios: Microsoft and real-world experience indicate that the Automatic (Intelligent) mode is optimal for the majority of use cases. It greatly reduces storage bloat while preserving the ability to recover recent and important versions. Make this your organization’s default unless you have a compelling reason not to. Many organizations have switched this on tenant-wide to curb runaway storage growth from versioning.

Use Manual Limits Where Necessary: There may be cases where a manual policy fits better. For example, a compliance-sensitive library might be required to keep all versions for at least 7 years, or conversely you might have a library of large video files where you only want the last 5 versions to save space. In such cases, set a specific manual limit (with or without expiration) appropriate to the scenario. For instance, you might configure 50 versions for a library with huge files, or “200 versions or 2 years” for a regulatory archive library. Document these deviations so you remember why they differ from the default.

Don’t Go Below 100 Versions/30 Days (UI Enforced Minimum): SharePoint Online’s interface won’t let you set extremely low limits – the rationale is to prevent administrators from accidentally setting a policy that could wipe out too much version history. Under the hood you can technically force lower values via APIs, but Microsoft strongly recommends against using less than 100 versions or trimming earlier than 30 days. Such aggressive limits could result in losing important recent edits and defeat the purpose of having version history. Stick to reasonable values that align with your recovery needs.

Educate Users on Versioning Impact: Ensure that site owners and users understand that versioning consumes storage. They should know that frequent saves (especially with AutoSave turned on) will generate many versions. This isn’t to discourage saving (the answer is not to turn off versioning!), but to reinforce why your organization manages versions the way it does. Users can also manually delete unnecessary versions from a file’s history if they know certain drafts or changes are not needed – though anything they delete manually goes to recycle bin for a period in case they made a mistake.

Leverage Reporting Tools: Take advantage of the Version Storage Usage report that Microsoft provides. This report can be run per site to see which libraries or files are consuming the most space via version history. It’s useful for identifying hotspots (e.g., a single file with 800+ versions taking 10 GB) and can guide you in applying proper limits or cleaning up. Before doing a large trim, always run the “what-if” analysis report to gauge impact.

Plan for Retention and Compliance: Be aware that retention policies and legal holds override version trimming. If a SharePoint site or an item is subject to a retention policy (through Microsoft Purview Compliance Center) or placed on eDiscovery hold, then no versions can be permanently deleted by any limit until that retention period is over. (Microsoft’s documentation explicitly states: “For items under a retention policy or hold, the document library’s versioning limits are ignored.”) This means your storage might continue to grow in those compliance scenarios. Best practice: coordinate with your compliance officers – if certain sites need infinite retention, you might leave their version limits looser (or just accept that storage will climb). Conversely, if you implement trimming, ensure it doesn’t conflict with any data retention requirements. The good news is that if a trim job encounters a version that is under retention/hold, it won’t delete it; it will tag an expiration date and then keep extending it until the hold is released, thereby not violating compliance.

Monitor Critically Important Documents: For content that is extremely sensitive or business-critical (e.g., an annually updated Policy document, or a legal contract file with tracked changes), you might want to keep more versions than usual or at least be very cautious with automated deletion. You can opt such libraries out of automatic trimming by setting a manual policy, or simply monitor their version history over time. Generally, Automatic mode is safe for even critical docs (since it preserves a broad range of history), but it’s wise to verify. If a particular version must be retained indefinitely (beyond what the algorithm would do), consider declaring the document a record or using a retention label on that version, which would prevent its deletion.

Conclusion

SharePoint’s Automatic Version History Cleanup (Intelligent Versioning) is a powerful feature that brings much-needed automation to version management. It keeps your SharePoint Online storage lean by removing redundant older versions while still providing a rich history of recent changes for recovery and audit purposes. By understanding how this feature works and following best practices — enabling it tenant-wide, adjusting specific libraries as needed, and considering organization-specific compliance requirements — IT administrators can significantly reduce storage costs and maintenance overhead.

With a sensible versioning strategy in place, you’ll ensure that users have the file history they need, when they need it, without letting “version sprawl” overwhelm your SharePoint environment. By configuring automatic cleanup and using the tools Microsoft provides (like reports and trim jobs), managing version history becomes a set-and-forget policy rather than a constant manual cleanup effort. This lets you and your users enjoy the benefits of versioning (easy recovery from mistakes, audit trails of changes) without the downsides of unchecked growth in your content databases.

With SharePoint Automatic Version History Cleanup, you can strike the right balance between data retention and storage efficiency – keeping your collaboration environments both agile and compliant.