Microsoft Mission Critical Blog articles

Managing SharePoint Online Version History at Scale to Recover Storage

mikeleemsft — Fri, 04 Sep 2026 18:11:38 GMT

Summay

SharePoint Online version history is one of those features that users love, but admins eventually have to govern.

Versioning protects users from accidental edits, bad changes, and overwritten content. It gives people a way to restore previous versions of documents without opening a ticket. That is a good thing.

But at tenant scale, version history can also consume a surprising amount of storage.

Highly active libraries, large Office files, frequent edits, migrations, sync activity, and years of retained versions can quietly add up. Eventually, admins need to answer a few practical questions:

How many file versions do we have?

How much storage are those versions consuming?

Which sites would benefit most from cleanup?

What policy should we apply going forward?

How much space could we recover before making changes?

How do we clean up versions safely across many sites?

That is why I built this tool: SPOVersions-ApplicationTools

The main script is: Apply-SPOVersions-Tool.ps1

Why version history planning matters

Microsoft's official guidance for SharePoint version storage makes an important point: version limits should support your organization's recovery objectives while also controlling storage consumption.

Microsoft documents this here:

Plan version storage for document libraries in SharePoint

The key idea is balance.

If you keep too many versions for too long, storage consumption can grow quickly. If you trim too aggressively, users may lose the ability to recover from accidental or unwanted changes.

A good version history strategy needs to consider both:

Goal	Why it matters
Recovery	Users need enough version history to restore useful prior versions.
Storage control	Admins need to prevent version history from consuming excessive quota.
Consistency	Tenant defaults and site-level policies should be understandable and repeatable.
Exceptions	Some business areas may need different version retention behavior.
Safe cleanup	Existing versions should be analyzed before deletion jobs are created.

The goal is not simply to delete versions. The goal is to understand version storage, set the right policy, and clean up intentionally.

What Microsoft recommends admins consider

Microsoft's documentation calls out three major planning questions:

Determine the default version history limits for new document libraries.

Identify exceptions needed from organization default limits.

Decide whether existing content should be updated to align with the organization limits.

That third point is where many admins need tooling.

It is one thing to configure a tenant default for new libraries. It is another thing to understand what already exists across hundreds or thousands of sites.

Existing content is often where the storage opportunity is.

Automatic vs. manual version limits

SharePoint provides different ways to manage version history.

Mode	What it does	Best fit
Automatic	SharePoint uses intelligent trimming based on version age. Recent versions are preserved more densely, while older versions are thinned over time.	Recommended default for many organizations that want Microsoft-managed optimization.
Manual with count and expiration	Admins specify how many major versions to keep and how long versions should remain.	Useful when you want more direct control and predictable cleanup.
Manual with count only	Admins keep a configured number of versions without age-based expiration.	Useful when a fixed restore count matters, but it can consume more storage.

Microsoft's guidance highlights that automatic trimming is designed to preserve high-value versions while optimizing storage. Manual policies can be useful, but they require careful planning because low limits or short expiration windows can affect restore scenarios.

That is why I wanted this tool to include reporting and What-If analysis before cleanup.

What the tool does

Apply-SPOVersions-Tool.ps1 is an interactive PowerShell tool built with PnP PowerShell to help admins manage SharePoint Online version policies across multiple site collections.

At a high level, the tool helps admins:

Audit current version policies across sites.

Apply automatic or manual version limits.

Review version policy status.

Generate version history reports.

Run What-If analysis to estimate storage recovery.

Create batch delete jobs for cleanup.

Monitor cleanup job status.

Set tenant-level defaults for new sites.

This makes it useful for both planning and implementation.

Why I built it as an admin workflow

Version cleanup should not start with deletion.

The safer workflow is:

Discover current settings.

Generate version history reports.

Estimate storage recovery.

Decide the right policy.

Apply policy to selected sites.

Queue cleanup jobs.

Monitor job status.

Repeat as part of governance.

That is how the tool is structured. It is not just a cleanup script. It is a planning and execution tool.

Site discovery options

The tool can work from a defined list of site URLs or discover sites automatically.

For larger tenants, I usually prefer a site list because it gives admins better control over rollout scope. You can start with a pilot group, validate results, then expand.

For smaller tenants or discovery scenarios, auto-discovery can be useful.

The tool supports:

SharePoint sites

OneDrive for Business sites

Exclusion of common system sites

Confirmation before processing discovered sites

That confirmation step matters. When a script can operate across many sites, admins should have a checkpoint before changes are made.

Key operations:

Get current version policies

Before changing anything, admins can retrieve current version policy settings for all selected sites.

This helps answer:

- Which sites already use automatic trimming?

- Which sites use manual limits?

- Which sites have inconsistent settings?

- Which sites may need exceptions?

This is useful for building a baseline.

Set version policies across sites

After planning, admins can apply version policies across site collections.

The tool supports:

- Automatic version trimming

- Manual major version limits

- Optional expiration after a specified number of days

Automatic trimming is useful when you want SharePoint to optimize storage intelligently. Manual limits are useful when the organization has a defined policy, such as keeping a specific number of major versions or expiring versions after a specific period.

Check version policy status

Policy changes may take time to apply.

The tool can check propagation status, so admins can verify whether pending changes are completed successfully.

This is important during staged rollouts because you want to confirm policy state before moving on to cleanup.

Generate version history reports

This is one of the most important parts of the workflow.

The tool can submit version history report jobs for each site. Those reports are written to a dedicated document library created by the tool:

Admin_SiteCollection_VersionReport_DONOTDELETE

Using a dedicated library avoids some of the issues that can happen when report files are written into normal user-facing document libraries.

The reports provide the raw data needed to understand version storage usage.

Run What-If analysis

This is the feature I think most admins should start with.

The What-If analysis downloads completed version history reports and simulates a version policy without making changes.

That means admins can estimate:

- Total versions analyzed

- Versions that would be deleted

- Total version storage

- Storage that could be recovered

- Percent of version storage that could be freed

- Per-site savings

This gives admins data before they commit to a cleanup policy.

For example, a What-If analysis might show that one site can recover a large amount of storage while another site has very little benefit. That helps prioritize action.

Create batchdeletejobs

Once the admin understands the impact and chooses a policy, the tool can create batch delete jobs to remove excess versions.

This is the destructive part of the workflow, so it should come after reporting and What-If analysis.

Admins can use automatic or manual cleanup criteria depending on the desired policy.

Important reminder: deleted versions are not recovered from the Recycle Bin. Treat batch delete jobs as permanent cleanup operations.

Monitor batchdeletejob status

After cleanup jobs are created, admins need to know whether they completed and how much storage was released.

The tool can check batch delete job status across sites, including completion state and storage released where available.

This closes the loop from planning to execution.

Tenant-level defaults vs. existing content

One important distinction in SharePoint version management is the difference between tenant-level defaults and existing content.

Tenant-level settings affect new sites or new libraries going forward. Existing sites and existing file versions may need additional steps.

Microsoft's guidance describes multiple approaches:

Scenario	What it means
Apply defaults to new libraries only	Least disruptive; existing settings and versions remain unchanged.
Update existing library settings without trimming versions	Future versions follow the new policy, but existing versions are not deleted.
Update settings and trim existing versions	Aligns current and future behavior but requires careful planning and cleanup jobs.

This tool helps with the third scenario, where admins need to understand and manage existing version storage across sites.

A practical rollout approach

For most tenants, I would not recommend starting with the entire environment.

A safer rollout looks like this:

Phase	Action
Phase 1: Baseline	Get current version policies across a small set of sites.
Phase 2: Report	Generate version history reports for pilot sites.
Phase 3: What-If	Estimate storage recovery under automatic or manual policies.
Phase 4: Decide	Choose default policy and identify exceptions.
Phase 5: Apply	Apply policy to selected pilot sites.
Phase 6: Cleanup	Queue batch delete jobs only after review.
Phase 7: Monitor	Check job status and storage released.
Phase 8: Expand	Repeat across additional sites or workloads.

This creates a defensible admin process instead of a one-time cleanup event.

Where this helps most

This tool is especially useful in tenants with:

Large document libraries

Heavy collaboration sites

Long-running project sites

Migration history

Large Office files with frequent edits

Storage pressure

Inconsistent site-level version policies

Need for tenant-wide governance reporting

It can also help organizations that want to move toward automatic trimming but need to understand the impact first.

Permissions and prerequisites

The tool uses PnP PowerShell and interactive authentication.

Typical prerequisites include:

PowerShell 5.1 or later

PnP.PowerShell 3.1.0 or later

SharePoint Administrator role

Entra ID app registration

Delegated SharePoint permissions required for version policy and report operations

Important: Ensure that the account running this tool has site collection admin rights on all sites

If you need to add your Admin account to all OneDrive and SPO sites, check out this scripts:

Add-GroupToSPOSites.ps1

Add-GroupToOneDriveSites.ps1

The repository README includes setup details and permission guidance:

Apply-SPOVersions-Tool README

As always, review permissions carefully before using any tenant-wide admin automation.

Safety notes

Version cleanup can permanently remove version history.

Before creating cleanup jobs:

Run reports first.

Run What-If analysis.

Start with pilot sites.

Review business requirements.

Identify exceptions.

Communicate expected behavior to site owners.

Keep exported results for audit and planning.

Avoid opening or modifying report CSV files while report jobs are still running.

The tool is designed to support safe planning, but admins still need to make policy decisions carefully.

Why this belongs in your governance toolkit

SharePoint storage management is not just about quota.

Version history touches:

User recovery expectations

Records and retention requirements

Collaboration patterns

Site lifecycle

Migration cleanup

Storage forecasting

Governance consistency

If version history is unmanaged, storage growth can feel mysterious. Once you report on it, it becomes something you can plan and control.

That is the point of this tool: help admins move from guessing to measuring, then from measuring to controlled action.

Final thoughts

SharePoint version history is valuable, but it needs governance at scale.

Microsoft provides flexible version storage options, including automatic trimming and manual limits. The challenge for admins is understanding what exists today and how much storage could be recovered before applying changes.

Apply-SPOVersions-Tool.ps1 helps with that process by combining policy discovery, report generation, What-If analysis, policy application, cleanup jobs, and job monitoring into a single admin workflow.

More Information

You can find the tool here: SPOVersions-ApplicationTools
Microsoft's official planning guidance is here: Plan version storage for document libraries in SharePoint
Suggested Reading for More Information: SharePoint Automatic Version History Cleanup (Intelligent Versioning)

AI Just Joined Your Code Reviews: GitHub Copilot Comes to Azure Repos

Shdsouza — Thu, 03 Sep 2026 13:31:18 GMT

Overview

Every engineering team runs into the same wall: the code is written, the tests are green, and then the pull request sits, waiting for a human to find time. Review is where good work can queue. GitHub Copilot code review for Azure Repos is currently in limited preview. It can take a first pass at a pull request and return feedback after a few moments, depending on the repository and the size of the change.

Here is how it works, how to switch it on, what it is genuinely good at, and where you should keep a human firmly in the loop.

GitHub Copilot acts as an automated reviewer on Azure Repos pull requests. When requested, it posts inline comments and one-click suggested edits before a human reviewer signs off.

Best fit: Early feedback on correctness, security, maintainability, and testability.
Human review retained: Domain, architecture, compliance, and required code-owner approvals.
Workflow: Treat findings like any pull request comment, validate, then fix, discuss, or resolve.

Review jobs run on an Azure Pipelines agent, so nothing is installed on developer machines.

The one thing to remember Copilot only comments. It never approves a pull request and never requests changes, so it cannot satisfy a required reviewer or block a merge on its own. Humans stay in control of the merge button, by design.

Prerequisites

Check these prerequisites before you begin.

Requirement	Detail
Platform & repo	Azure DevOps Services with an Azure Repos Git repository. TFVC is not supported.
Permissions	Project Collection Administrator (org), Project Administrator (project), and repository owner/administrator (repo).
Billing	An Azure subscription linked to your organization. Usage is metered through Azure Cost Management. For details, see Billing.
Compute	A supported Microsoft-hosted Azure Pipelines agent pool, or a Managed DevOps Pool using the latest Ubuntu Server image. Windows images and self-hosted agent pools are not supported.

Enable Copilot code review

Access is layered across three scopes; enable the organization first. Use Enable all at the org or project level to cover everything at once.

Organization: Organization settings > Repos > Repositories > Allow repositories in this organization to use Copilot code review = On.
Project (optional): Project settings > Repos > Repositories, enable or restrict per project.
Repository: select the repo > Settings tab > Enable Copilot code review for pull requests in this repository = On.

To verify, open any pull request - GitHub Copilot should appear in the Reviewers list.

Configure reviews

Default review effort

Effort	When to use	Relative cost
Lite	Routine changes, small fixes, docs, low-risk updates.	Fewer tokens
Balanced	Complex, cross-cutting, or security-sensitive changes.	More tokens

Naming note: you may see Lite/Balanced or Low/Medium/High(not yet released); higher effort costs more.

Automatic reviews & instructions

Automatic review: configure a policy at project or repository level to review new pull requests on creation.
Custom instructions: tailor feedback via .github/copilot-instructions.md or .azuredevops/copilot-instructions.md on the PR's target branch, or org/project settings. Using custom instructions to unlock the power of Copilot code review - GitHub Docs

Request and use a review

Next to GitHub Copilot in Reviewers, select Request (or pick an effort from the menu).
Wait for Review completed, then validate each comment before acting.

Comment-only: Copilot never approves or requests changes, so it doesn't satisfy required reviewers or block merges.
Interactive: reply, resolve, or hide comments. Copilot doesn't read replies or follow up.
Re-review / cancel: it doesn't re-review after new commits, select Request again; an in-progress review can be cancelled. To automatically request re-reviews on every push, enable automatic code review. For more information, see Configuring code review by GitHub Copilot.

Scope and limits

Copilot reviews many languages but skips dependency, log, SVG, and vendor/generated files (package-lock.json, yarn.lock, etc.); admins can exclude more paths. Results depend on the difference, so runtime, architecture, and business rules may be missed; always validate.

Requirement / limit	Value
Pull request status	Active, no merge conflicts
Repository size / changed files	≤ 10 GB / ≤ 100 files
Concurrent reviews (org / user / PR)	5 / 2 / 1

Billing

Reviews are token-based (1 GitHub AI credit = US$0.01), charged to the linked Azure subscription as a separate meter in Azure Cost Management, not from Copilot plan credits. Effort, PR size, and instructions drive cost; charges can take 48 hours to appear.

Monitor: Cost Management > Cost analysis, filter by product; project tags enable per-project reporting.
Budget alerts: filter to product “GitHub Copilot for AzDO”. Alerts notify only; they don't stop reviews. Pilot on a repo or two to estimate cost before scaling.

Github Copilot billing

Recommended practices

Keep pull requests small; retain branch policies and required human reviewers.
Fix valid findings; reply to explain false positives. Keep secrets out of source control.
Encode recurring standards in custom instructions, tests, and linters; choose effort deliberately.

Troubleshooting

Symptom	Check
Copilot not offered as reviewer	Enablement, permissions, linked subscription/billing, repo type, Preview-features opt-in.
Review stalls or fails	Service health and agent pool; failed runs post a log link. Request again.
A file wasn't reviewed	Excluded type, content-exclusion path, or evaluated as low-risk.

Visual walkthrough

Step-by-step: turn it on

Enablement is layered across organization, project, and repository settings. Enable the organization first, then decide whether to enable all projects and repositories or configure them selectively.

Step 1 - Enable at the organization

Go to Organization settings > Repos > Repositories, and under GitHub Copilot code review turn on Allow repositories in this organization to use Copilot code review. Select the Compute pool that will run the review jobs and, if needed, add organization-wide custom instructions.

Organization settings: the master toggle, Compute pool, and organization-wide review instructions.

Step 2 - Enable the repository and set the default effort

In Project settings > Repos > Repositories, pick the repository, open the Settings tab, and turn on Enable Copilot code review for pull requests in this repository. Choose a default effort level - more on that below.

Repository settings: Copilot enabled for pull requests, with its default review effort.

Step 3 - Request a review on a pull request

Open any pull request. GitHub Copilot now appears in the Reviewers panel. Select Request, or choose a specific effort from the menu. After the review finishes, the status changes to Review completed. Prefer it hands-off? Configure the policy that requests a Copilot review whenever a new pull request is created.

Copilot listed among the reviewers, with its completed review posted to the timeline.

Step 4 - Review and resolve the findings

Copilot posts each finding inline on the line it applies to and, where possible, includes a one-click suggested change. Validate each finding, then apply the suggestion, reply, react, or resolve the thread. Copilot does not read replies or follow up, and it does not automatically re-review after new commits, select Request again for a fresh review.

Inline findings on the changed code - each ready to validate, address, discuss, or resolve.

Choosing a review effort

The effort level controls the depth of the review and the amount of token usage.

Lite - the default option for fast, targeted feedback on common issues such as bugs, security vulnerabilities, and style inconsistencies.
Balanced - routes the pull request to a higher-reasoning model for longer analysis of complex logic, security-sensitive code, and subtle interactions. It generally consumes more tokens.

The documented options are Lite and Balanced. Higher effort generally analyzes more context and consumes more tokens; pull request size and custom instructions can also affect usage.

The honest scorecard

✓ Strengths	⚠ Watch-outs
Fast first-pass feedback after a few moments, depending on the repository and the number of changes.	Comment-only; it never blocks a merge, so you still need required human reviewers.
Catches common issues such as bugs, security vulnerabilities, and style inconsistencies.	It is not guaranteed to spot every problem and can make mistakes; always validate its feedback with human review.
Inline comments and, where possible, one-click suggested changes for code in any language.	Excludes dependency, log, SVG, vendor, generated, and other documented file patterns.
Works on demand or automatically on Pull requests.	Consumes tokens; an ongoing cost that scales with effort and PR size.
Tunable with organization, project, and repository-level custom instructions.	Preview: no SLA, may change, and data residency may not match your Azure DevOps boundaries.

The bottom line

Copilot code review is not here to replace your reviewers - and the product behavior makes that clear. It can take the first, more mechanical pass at a pull request so issues surface earlier and people can focus on context, design, and business impact. In a busy repository, that can reduce review friction without giving up human control.

Sources and further reading

Microsoft Learn - Get started with Copilot code review for pull requests (updated August 27, 2026): https://learn.microsoft.com/en-us/azure/devops/repos/git/copilot-code-reviews?view=azure-devops

GitHub Docs - About GitHub Copilot code review: https://docs.github.com/en/copilot/concepts/agents/code-review

GitHub Docs - Files excluded from GitHub Copilot code review: https://docs.github.com/en/copilot/reference/review-excluded-files

The new sharing experience is coming to SharePoint Online: What admins need to know

mikeleemsft — Fri, 04 Sep 2026 18:12:24 GMT

Microsoft is introducing a new sharing experience for Microsoft 365 that is designed to simplify collaboration and make link-based access easier to manage. At the center of this update is the hero link: a single sharing link for a file that is reused across common sharing actions such as copying a link, sharing by email, or using the file URL.

Roadmap ID: 492622
Rollout start: June 2026

This post is intended for Microsoft 365 and SharePoint Online administrators who want to understand what is changing in the new sharing experience, how the interface differs from the current experience, and how to prepare end users as rollout continues.

Meet the hero link

Hero sharing visual

The hero link is the core change in the new sharing model. Each file uses a single sharing link that controls access, whether a user selects Copy link, shares by email, or copies the file URL from the browser.

This reduces the need to create and manage multiple links for the same item and makes sharing behavior more consistent.

What changes in the sharing interface

The screenshots below highlight some of the most noticeable differences between the classic sharing experience and the new sharing experience. They are intended to help admins and support teams quickly understand where familiar actions have moved and how behavior changes in the new model.

Sharing UI

Classic sharing experience: More settings are exposed up front, which gives flexibility but can feel busier and harder to scan.

Classic SharePoint Online sharing UI

New sharing experience: The experience is simpler and more guided, helping users share faster with less decision friction.

New sharing UI

Sharing settings

Classic sharing experience: Link options and permission settings can feel more fragmented, requiring users to interpret multiple choices.

Classic link settings

New sharing experience: Settings are more streamlined around the hero link, making access management easier to understand and reuse.

New link settings

Link scope

Classic sharing experience: Classic sharing relies more on separate link choices and less flexible scope adjustments.

Classic link scope options

New sharing experience: The new link scope experience lets users change permission scope while continuing to use the same link, making sharing more flexible without creating separate links.

New link scope options

Expiration links

Classic sharing experience: Creating links with expiration dates is built into the main link settings, so users can configure expiration as part of the standard sharing flow.

Classic expiration link settings

New sharing experience: Creating expiration links is not yet fully incorporated into the main flow. Users must click the gear icon and choose Create new sharing link to create a link with expiration settings.

Menu option to access new link settingsNew expiration link settings

Stop sharing

Classic sharing experience: Stop sharing is accessed from Manage Access. Stopping sharing removes all unique permissions from the item and causes it to inherit permissions from the top-level object again.

Classic stop sharing options

New sharing experience: Stop sharing is accessed from Sharing settings. Stopping sharing removes all access from the item and leaves only the Owners group in the permission list for that object.

New stop sharing options

Manage Access

Classic sharing experience: Manage Access separates sharing details into different tabs for People, Groups, and Links. From there, users can review existing access, remove members or links, and open advanced settings for deeper permission management.

Classic Manage Access experience

New sharing experience: Manage Access is a single unified view where users can see people, groups, and links together. This makes it easier to add or remove access without switching between separate tabs, and advanced settings are available from the gear icon.

New Manage Access experience

Advanced settings

Classic sharing experience: Advanced settings are reached from Manage Access by clicking the More options button.

Classic advanced settings

New sharing experience: Advanced settings are available from the main sharing link properties by clicking the gear icon.

New advanced settings

Key takeaways for end users

For end users, the main change is not what sharing can do, but how sharing is presented and managed.

The new experience is built around a single reusable sharing link, so actions such as Copy link, sharing by email, and using the file URL are more consistent than before. Users will also notice that some familiar actions have moved. Manage Access is now a unified view, advanced settings are accessed from the gear icon, and tasks such as creating expiration links follow a slightly different flow than they do in the classic experience.

The result is a simpler interface that may require a short adjustment period for users who are familiar with the classic layout.

First-run experience

To help users adjust to the new sharing model, Microsoft has added a first-run experience that appears the first time a user opens the Share dialog. It introduces the main parts of the new experience and explains how the hero link works.

The first-run experience guides users through the core components of the dialog and helps them learn how to get the most out of the hero link.

If users want a refresher later, they can click the ellipsis and select View tutorial to launch the walkthrough again.

First-run experience View tutorial optionFirst-run experience tutorial walkthrough

Customer communications

Before rollout reaches your tenant, admins should review the updated sharing flow, validate internal help content, brief support teams on where key controls have moved, and prepare a short user communication for frequent sharers and site owners.

Here is a sample communication you can adapt for your organization:

Hello team,

We want to let you know about an upcoming update to the SharePoint Online and OneDrive sharing experience. This update introduces a simpler sharing experience centered around a single sharing link for each file, helping make sharing more consistent and easier to manage. Existing links and permissions will continue to work, so no action is required for files that have already been shared.

As part of this change, some sharing options will appear in different places than they do today. For example, users may notice updates to Sharing settings, Manage Access, Stop sharing, and how expiration links are created. The overall goal is to simplify the experience, reduce link sprawl, and make access management easier to understand.

Before this reaches our tenant, we recommend that site owners, frequent content sharers, and support staff review the updated experience so they are familiar with the new layout and behavior. We will share more guidance and screenshots before rollout is complete.

If you have questions after the new experience appears, please contact your SharePoint or Microsoft 365 support team.

More information

This new experience was demonstrated during the SharePoint 25th Birthday event. You can watch the full event playlist here: SharePoint at 25 | Birthday Celebration Event.

You can also read the Product Group blog post: Simple, Smart, and Secure: The next step in sharing files in Microsoft 365.

Known issues and current limitations

Word, Excel, and PowerPoint apps on Mac currently use an older version of the Share dialog and therefore do not yet have the new experience. This is expected to roll out shortly.

FAQ

What’s changing with sharing?

SharePoint Online and OneDrive are moving to a single canonical sharing link per file or folder. Whether a user selects Copy link, shares by email, or copies the file URL, the same link is reused instead of creating multiple separate links for the same item.

What problem does it solve?

Today, sharing can create a different URL almost every time. The new model replaces that sprawl with one stable link per item, so links embedded in emails, chats, and documents stop breaking when you adjust who has access.

How does the new sharing link change my governance model?

There is less link sprawl to govern. One canonical link per item replaces multiple parallel links with independent lifecycles. Policy, audit, and lifecycle decisions now attach to a single object per file or folder instead of a fan-out of link variants.

Will any existing links or permissions break during rollout?

No. Existing sharing links and permissions will continue to work during rollout, so users do not need to reshare content or recreate access that is already in place.

What if a link leaks?

You can invalidate it. The old URL stops working, and a new sharing link is issued for the file. Direct permissions on the item are not touched; only the link is rotated.

How does this interact with information barriers?

Information barriers continue to prevent sharing across segmented groups. The new sharing link cannot be used to grant access in violation of an information barrier policy.

Why does the new sharing link matter for Copilot?

Copilot grounds its answers in content that a user already has access to. The more intentional your sharing posture is, the easier it is to understand who can access what content and the lower the risk of oversharing.

By reducing link sprawl and consolidating sharing around a single canonical link per item, the new model gives admins a clearer view of access that supports Copilot readiness.

Does the new sharing link reduce oversharing risk for Copilot?

Indirectly, yes. Today, multiple parallel links per item with independent lifecycles make it hard to see and remediate oversharing. One canonical link with explicit scope and re-share controls makes over-permissioning easier to detect and clean up before turning on Copilot more broadly.

Final takeaway

The new sharing experience simplifies how links are created, reused, and governed in SharePoint Online and OneDrive. For admins, the main preparation work is straightforward: understand the hero link model, review the UI differences called out in this post, and proactively communicate the changes to users before rollout reaches your tenant.

Existing links and permissions continue to work, but the user experience will change enough that a short readiness plan is worth putting in place now.

Document Contributors: Mike Lee | Mariel Williams

Your on-call rotation has a new member: 10 production incidents, end to end, with Azure SRE Agent

zmustafa — Fri, 28 Aug 2026 16:14:08 GMT

TL;DR
1. Why the "investigation" half is the real prize
2. What Azure SRE Agent actually does
3. Anatomy of an agent-run incident
4. Build the demo lab
5. Set your guardrails before your first incident
6. The ten use cases
7. The ITSM integration model
8. Reality check: where you have to build
9. Approval and autonomy policy
10. Cross-cutting security controls
- What the platform gives you for free
- The audit trail
11. A 30/60/90 pilot that survives contact with your CAB
- The best first five candidates
- What to measure in week one
12. Measuring whether it's actually working
13. Resources
Closing

What this post is. A hands-on, reproducible walkthrough of ten real production failure modes — App Service, AKS, Azure SQL, Cosmos DB, VMs, VM Scale Sets, Application Gateway, and Service Bus — each one driven end to end by Azure SRE Agent: detection, hypothesis-driven investigation, ITSM ticketing, bounded remediation, recovery validation, and follow-up records.

What this post is not. A claim that you install SRE Agent and all of this happens on day one. Every workflow below needs telemetry, scoped RBAC, a response plan, an approved action surface, and an ITSM integration. I'll be explicit about which parts are documented product behavior and which parts you have to wire yourself — because that distinction is the difference between a demo that works on stage and one that works at 3 AM.

TL;DR


The pattern	Alert → agent investigates read-only → agent opens/updates the ticket with evidence → agent proposes a bounded action → human approves → agent executes → agent validates recovery → agent files the follow-up record
The unlock	Not "AI fixes prod." The unlock is that investigation — the 20 minutes of tab-switching between Azure Monitor, App Insights, deployment history, and Activity Log — is fully automated and consistent, and the fix arrives pre-justified with an evidence chain
The guardrail	Read-only automatic. Ticketing automatic. Production writes in Review mode. Guest-OS work through fixed-purpose runbooks, never a shell prompt
The reality check	SRE Agent hard-blocks `delete`/`remove` and all `az keyvault` commands, respects Azure management locks, and only one incident platform can be active at a time. Several patterns in this post need a custom tool or MCP server to complete the loop
Time to first value	One resource group, one alert rule, one response plan. You can reproduce use case #1 in an afternoon

1. Why the "investigation" half is the real prize

Every conversation about AI in operations goes straight to remediation. "Will it restart my app?" That's the least interesting question, and it's the one with the most downside risk.

Think about what actually consumes the minutes during a Sev1. The alert fires. Someone acknowledges. Then:

Open Azure Monitor, confirm the metric is real and not a probe artifact
Open Application Insights, find the dominant exception
Open the deployment pipeline, find what shipped and when
Open Activity Log, check whether someone changed configuration
Open Resource Health, rule out a platform incident
Open the other environment, confirm the previous version is healthy
Assemble all of that into a sentence a human can act on

That's twenty minutes of context assembly performed by a tired human, differently every time, with quality that depends entirely on who happens to be on call. It is the single most automatable part of incident response and the part nobody automates, because scripts can't reason about which of six hypotheses fits the evidence.

This is exactly what root cause analysis in SRE Agent is designed for. The agent doesn't grep logs — it forms hypotheses and invalidates them:

HYPOTHESIS 1: Recent deployment broke something ├─ Checked: Last deployment was 3 days ago ├─ Evidence: Error rate stable until 30 minutes ago └─ Result: INVALIDATED HYPOTHESIS 2: Database overloaded ├─ Checked: Azure SQL metrics (CPU, DTU, connections) ├─ Evidence: DTU at 98%, query duration 4x normal ├─ Traced: SELECT * FROM orders WHERE... taking 8.2s └─ Result: VALIDATED ROOT CAUSE: Orders table missing index on customer_id column. Query plan shows full table scan on 2.1M rows. RECOMMENDED ACTION: Add index on orders.customer_id Similar fix applied in INC-2341 (3 weeks ago)

That last line — recalling a similar incident from three weeks ago — is the compounding part. Every thread produces a session insight: symptoms observed, steps that worked, root cause, and pitfalls to avoid. Thirty minutes after a thread goes quiet, the agent indexes those learnings. Next time the same resource misbehaves, that history surfaces first.

So the framing for the rest of this post: remediation is the punchline, but investigation is the product. Every one of the ten use cases below has a large read-only phase you can turn on tomorrow with zero write permissions, and a small write phase you should gate behind approval for a long time.

2. What Azure SRE Agent actually does

Before the use cases, here is the honest capability map, drawn from the product documentation rather than from a keynote.

The three primary use cases

Use case	What it means
Automate incidents	Alert fires → agent queries monitoring tools, correlates signals across systems, identifies probable root cause, proposes mitigations
Automate scheduled workflows	Proactive health checks, compliance sweeps, and routine tasks on a schedule, with results routed to your incident platform or notification channel
Investigate and advise	Natural-language questions — "what changed in the last hour?" — answered with grounded citations

The five extension primitives

Everything you customize sits in one of five buckets:

Primitive	What it is	When you reach for it
Skills	Procedural guidance (`SKILL.md`) plus optional attached tools; auto-loaded when relevant	Team troubleshooting runbooks that should also execute
Subagents / custom agents	Purpose-built specialists invoked via `/agent` or routed to by a response plan	A `DatabaseExpert` that owns every SQL incident
Python tools	Custom logic, transformations, API calls	Anything that needs code, e.g. writing to the ServiceNow Table API
MCP servers	40+ managed connectors (Datadog, New Relic, Splunk, Elastic, Dynatrace…) plus any custom MCP tool	Bringing your non-Azure telemetry and your ITSM write surface into the loop
Agent hooks	Event-triggered automations at `Stop` and `PostToolUse`	Policy enforcement, audit emission, blocking risky commands

Six generic subagents ship built in — Explore, Plan, CodeReview, Bash, Verification, GeneralPurpose — and the agent can parallelize investigation, planning, review, shell, and verification work across them.

A permission gate sits in front of all five primitives and evaluates every proposed tool call before it runs.

Integrations you can assume

Category	Supported
Monitoring	Azure Monitor (metrics, logs, alerts, workbooks), Application Insights, Log Analytics, Grafana
Incident platforms	Azure Monitor Alerts, PagerDuty, ServiceNow — only one active at a time
Source control / CI	GitHub (repos, issues), Azure DevOps (repos, work items)
Data	Azure Data Explorer (Kusto), MCP servers
Comms	Slack, Microsoft Teams

⚠️ Design constraint worth internalizing early. Only one incident platform can be active at a time, and switching disconnects the current one. If your org runs PagerDuty for paging and ServiceNow for records of truth, you must pick which one the agent is bound to and reach the other through a connector or custom tool. Every use case below assumes ServiceNow is the bound platform.

Skills vs. custom agents vs. knowledge files

The three are constantly confused. This table settles it:

	Skills	Custom agents	Knowledge files
Access	Automatic when relevant	Explicit (`/agent`) or routed by response plan	Automatic search
Tools	Can attach	Has its own	None
Context	Uses thread context	Shares thread context (no clean slate)	Reference only
Best for	Team procedures with execution	Domain specialists	Runbooks, architecture docs

Practical limits: a maximum of five concurrent active skills (oldest auto-unloads), knowledge base uploads up to 16 MB per file, and custom agent knowledge bases up to 1,000 files.

3. Anatomy of an agent-run incident

Every use case in section 6 is an instance of this one shape. Learn it once.

[[INCIDENT_FLOW_IMAGE]]

Five properties make this shape safe, and they're worth stating as rules:

The read phase has no blast radius. Turn it on everywhere, immediately, with Reader.
One action per incident. Not "roll back and scale and restart." One bounded, reversible step, then re-measure.
The proposed action is always the smallest reversible one. Swap a slot, don't redeploy. Bump one service tier, don't resize the cluster.
Validation is a first-class phase, not a vibe. Define the metric, the threshold, and the duration before you approve.
Technical recovery ≠ business recovery. Use case #10 makes this painfully clear.

4. Build the demo lab

Everything below runs in a single throwaway resource group. Nothing here should touch a subscription you care about.

🧪 Lab hygiene. Create it, demo it, delete it. az group delete -n rg-sre-agent-demo --yes --no-wait when you're done. Several of these use cases deliberately break things.

4.1 Resource group and agent

LOC=eastus2 RG=rg-sre-agent-demo az group create -n $RG -l $LOC

Create the SRE Agent from the Azure portal and point it at $RG. Three things are created for you automatically:

an Application Insights instance (this is where your audit trail lands),
a Log Analytics workspace,
a user-assigned managed identity (UAMI) — the identity every action runs as.

4.2 Choose a permission level deliberately

At creation you pick one of two levels:

Level	Grants	Use it when
Reader	Core monitoring roles + resource-type reader roles. Prompts for temporary elevation via on-behalf-of when it needs to act	Start here. Production. Weeks 1–4 of any pilot
Privileged	Core monitoring roles + resource-type contributor roles	Non-production, or after a proven pilot

Regardless of level, these are always assigned:

Role	Scope	Why
`Reader`	Resource group	See resources and properties
`Log Analytics Reader`	Resource group	Query logs and workspaces
`Monitoring Reader`	Resource group	Read metrics
`Monitoring Contributor`	Subscription	Acknowledge and close Azure Monitor alerts

Grant additional access explicitly and narrowly:

SUB=$(az account show --query id -o tsv) AGENT_MI=<agent-managed-identity-principal-id> # Read everywhere you want visibility az role assignment create \ --assignee $AGENT_MI \ --role "Reader" \ --scope "/subscriptions/$SUB" # Write ONLY where you intend the agent to act az role assignment create \ --assignee $AGENT_MI \ --role "Website Contributor" \ --scope "/subscriptions/$SUB/resourceGroups/$RG/providers/Microsoft.Web/sites/app-checkout-demo"</agent-managed-identity-principal-id>

📌 A sharp edge. You can't remove individual permissions from an agent — only entire resource groups. Removing a resource group from the agent's scope revokes all access to it. Plan your resource group boundaries as your blast-radius boundaries, because that's exactly what they are.

4.3 Connect ServiceNow

Use a dedicated, least-privileged integration user — not a shared admin account.

Method	When	What you need
Basic auth	Quick setup, PDI, testing	Username + password, `itil` or `admin` role
OAuth 2.0	Production	ServiceNow OAuth app (client ID + secret); register redirect `https://logic-apis-{region}.consent.azure-apim.net/redirect`

Scope the connection so you don't drown:

Assignment group — essential on a shared enterprise instance
Priority — Critical through Planning
Category

Scanner defaults worth knowing:

Setting	Value
Scan interval	1 minute
Incidents per page	20
Max incidents per cycle	220 (11 pages)
Initial lookback	30 days

Setup performs a real connectivity check by fetching an actual incident, so credential and endpoint mistakes surface immediately instead of six hours later when nothing syncs.

🚨 Delete the quickstart plan. Connecting an incident platform auto-creates a quickstart_handler response plan that runs in fully autonomous mode across all impacted services. If you then build your own plans, incidents get routed twice or to the wrong agent. Go to Builder → Incident response plans → Table view and delete it before you do anything else.

4.4 Connect deployment correlation

Half the use cases below hinge on "what shipped three minutes before the spike." That correlation requires a source control connector:

GitHub — repositories and issues
Azure DevOps — repos and work items

Connect one. Without it, the agent can still read Azure Activity Log and deployment history, but it can't reach commits, PRs, or work items.

4.5 Create the demo resources

# 1 · App Service with a staging slot (use cases 1) az appservice plan create -g $RG -n plan-demo --sku P1V3 --is-linux az webapp create -g $RG -p plan-demo -n app-checkout-demo --runtime "DOTNETCORE:8.0" az webapp deployment slot create -g $RG -n app-checkout-demo --slot previous # 2 · AKS (use case 2) az aks create -g $RG -n aks-demo --node-count 3 --generate-ssh-keys --enable-addons monitoring # 3 · Azure SQL (use case 3) az sql server create -g $RG -n sqlsrv-sre-demo -u sqladmin -p "<use-a-generated-password>" az sql db create -g $RG -s sqlsrv-sre-demo -n db-customer --service-objective S1 # 4 · Cosmos DB (use case 4) az cosmosdb create -g $RG -n cosmos-sre-demo az cosmosdb sql database create -g $RG -a cosmos-sre-demo -n catalog az cosmosdb sql container create -g $RG -a cosmos-sre-demo -d catalog \ -n products --partition-key-path /category --throughput 400 # 5–7 · VMs (use cases 5, 6, 7) az vm create -g $RG -n vm-payments-linux --image Ubuntu2204 --size Standard_B2s --generate-ssh-keys az vm create -g $RG -n vm-claims-win --image Win2022Datacenter --size Standard_B2ms \ --admin-username azureadmin --admin-password "<use-a-generated-password>" # 8 · VM Scale Set (use case 8) az vmss create -g $RG -n vmss-api-demo --image Ubuntu2204 --instance-count 12 \ --upgrade-policy-mode automatic --generate-ssh-keys # 10 · Service Bus (use case 10) az servicebus namespace create -g $RG -n sb-sre-demo --sku Standard az servicebus queue create -g $RG --namespace-name sb-sre-demo -n order-events \ --enable-dead-lettering-on-message-expiration true</use-a-generated-password></use-a-generated-password>

Install the Azure Monitor Agent on the VMs and associate a data collection rule — without guest telemetry, use cases 5, 6, and 7 have nothing to detect.

4.6 One custom agent per domain

Don't build a single mega-agent. Build specialists and let response plans route to them:

Custom agent	Owns	Attached tools
`DeploymentAnalyzer`	App Service, AKS, anything release-correlated	`RunAzCliReadCommands`, GitHub connector, Kusto
`DatabaseExpert`	Azure SQL, Cosmos DB	`RunAzCliReadCommands`, Kusto, read-only SQL diagnostics tool
`GuestOSResponder`	VM / VMSS / IIS	`RunAzCliReadCommands`, fixed-purpose runbook tools only
`NetworkPathExpert`	Application Gateway, NSG, DNS, TLS	`RunAzCliReadCommands`
`IntegrationExpert`	Service Bus, Event Hubs	`RunAzCliReadCommands`, Kusto

A custom agent definition is small:

name: database_expert system_prompt: | You are a database specialist for this estate. Analyze query performance, diagnose connection and saturation issues, and recommend the smallest reversible mitigation. Never propose schema changes, index changes, plan forcing, or session termination — those are DBA-owned and require a change record. handoff_description: Handles Azure SQL and Cosmos DB troubleshooting tools: - execute_kusto_query - RunAzCliReadCommands allowed_skills: - azure-sql-saturation-runbook - cosmos-throughput-runbook

Note what that system_prompt is really doing: it is narrowing the action space. Half of production safety with an agent is telling it, in plain English, which categories of fix are off the table.

4.7 Response plans

Create one plan per domain, all starting in Review:

Plan	Filter	Custom agent	Mode
`appsvc-p1`	Priority 1 + 2, service `checkout`	`DeploymentAnalyzer`	Review
`aks-p1`	Priority 1, service `orders-api`	`DeploymentAnalyzer`	Review
`db-critical`	Priority 1 + 2, service `customer-api`	`DatabaseExpert`	Review
`vm-guest`	Priority 1 + 2, title contains `disk`, `memory`, `CPU`	`GuestOSResponder`	Review
`network-p1`	Priority 1, title contains `502`, `backend`	`NetworkPathExpert`	Review
`integration-p1`	Priority 1, service `fulfillment`	`IntegrationExpert`	Review

Filters available: severity/priority (multiselect), impacted service, incident type, and title contains. Plans can be turned off without deleting them, which is exactly what you want during maintenance windows.

5. Set your guardrails before your first incident

If you read only one section of this post, read this one. The guardrails are not paperwork; they are the reason this is deployable.

5.1 Run modes

Mode	Behavior	Default
Review	Agent proposes; you approve or deny	Agent-level default
Autonomous	Agent executes immediately and reports	Per-plan and per-task default

Two subtleties that bite people:

Run modes are set per response plan and per scheduled task, not globally. The agent-level setting is only a fallback. And per-plan the default is Autonomous — so if you don't set it, you get autonomy.
Review mode shows Approve/Deny only for Azure infrastructure operations (Azure CLI, ARM writes). Sending an email, posting to Teams, or querying an external source proceeds based on the agent's reasoning. To gate those, you need hooks or tool access policies.

Only users holding the SRE Agent Administrator role can approve. Standard users cannot, and personal Microsoft accounts can't authorize on-behalf-of at all — it requires a work or school (Entra ID) account.

5.2 What the product blocks for you

These are enforced at the command level, independent of your RBAC:

Guardrail	Behavior
Delete operations	The agent never runs `delete` or `remove` commands. It returns an error pointing you at the portal
Key Vault	All `az keyvault` commands are blocked, to prevent credential exposure
Management locks	Resources with `ReadOnly` locks can't be modified, regardless of permissions or run mode
Subscription validation	Subscription IDs are validated as well-formed GUIDs before execution

💡 Use the delete block architecturally. Put a ReadOnly management lock on anything that must never change during an incident — your Key Vaults, your production databases, your golden images. That lock is respected before every modification, which gives you a control that survives RBAC drift and misconfigured response plans.

5.3 Hooks: the guardrail you write yourself

Two events are supported — Stop (agent about to return a final response) and PostToolUse (a tool finished). Hooks run as an LLM prompt or a sandboxed command script.

Here is the single most useful hook for this entire post — a deterministic policy gate on shell execution:

hooks: PostToolUse: - type: command matcher: "Bash|ExecuteShellCommand" timeout: 30 failMode: block script: | #!/usr/bin/env python3 import sys, json, re context = json.load(sys.stdin) command = context.get('tool_input', {}).get('command', '') dangerous = [ r'\brm\s+-rf\b', r'\bsudo\b', r'\bchmod\s+777\b', r'\bmkfs\b', r'\bdd\s+if=', r'\btruncate\b', r'\bDROP\s+TABLE\b', ] for pattern in dangerous: if re.search(pattern, command, re.IGNORECASE): print(json.dumps({"decision": "block", "reason": f"Blocked by policy: {pattern}"})) sys.exit(0) print(json.dumps({"decision": "allow"}))

And a Stop hook that refuses to let the agent declare victory without evidence:

hooks: Stop: - type: prompt model: ReasoningFast timeout: 30 prompt: | Review the agent's final response. $ARGUMENTS It is only acceptable if it contains ALL of: 1. The specific resource acted upon (full name or resource ID) 2. Metric values BEFORE and AFTER the action 3. The observation window over which recovery was confirmed 4. Who approved the action and at what UTC timestamp Respond with: {"ok": true} {"ok": false, "reason": "<what is="" missing="">"}</what>

Hook mechanics you'll want on hand:

Setting	Default	Range / notes
`timeout`	30s	1–300s
`failMode`	`allow`	`allow` or `block`
`maxRejections`	3	1–25; prompt-type `Stop` hooks only
`matcher`	—	Regex, anchored `^(pattern)$`, case-sensitive; `*` matches all
Script size	—	64 KB max
Shebangs	—	`#!/bin/bash`, `#!/usr/bin/env python3`

⚠️ The footgun: for Stop hooks, a rejection without a reason field is treated as approval. Always populate reason.

Agent-level hooks and custom-agent-level hooks both run when both match; agent-level fires first.

5.4 The recommended production policy

This is the policy I'd put in front of a change board:

Allow read-only investigation automatically, everywhere.
Allow automatic incident creation and work-note updates — after sanitization.
Use Review mode for all production remediation.
Require human approval for every production write.
Use fixed-purpose runbooks instead of unrestricted VM shell access.
Require separate approval for destructive or data-affecting actions.
Initially require human confirmation before resolving an incident.

6. The ten use cases

Each use case follows the same seven-part structure so you can skim to the one you're firefighting:

What happened → How Azure Monitor detected it → How the agent found the cause → Agent action table → Recovery note → Try it yourself → Guardrails

In the action tables, Type is one of:

Type	Meaning
`Read`	No blast radius. Safe to run automatically
`Decision`	Agent reasoning or an approval gate. No resource change
`Write (ITSM)`	Ticket create/update. Sanitized
`Write (Azure)`	Azure control-plane change. Approval required in production
`Write (Guest OS)`	Inside the VM, via a fixed-purpose runbook. Approval required
`Write (K8s/DevOps)`	Cluster or pipeline change. Approval required
`Validation`	Post-action measurement against defined thresholds

Use case #1 — App Service: HTTP 500 after deployment

The one you should build first. Clean trigger, bounded action, unambiguous validation.

What happened

Release 2026.03.12.4 was deployed to a production checkout App Service. The new code referenced an application setting that was never defined in the production slot. The application started fine — which is what makes this class of failure nasty — but every checkout operation that touched that setting returned HTTP 500.

How Azure Monitor detected it

Azure Monitor and Application Insights fired on a composite condition, not a single metric:

HTTP 5xx rate above the operational threshold
Failed availability tests
Increased application exceptions
Increased dependency failures
Request latency above baseline

P1 – Checkout App Service returning elevated HTTP 500 responses

How the agent found the cause

Identified the affected App Service and slot.
Queried HTTP response and latency metrics.
Queried Application Insights failed requests and exceptions.
Identified the missing-setting exception as the dominant failure.
Reviewed deployment history and Azure Activity Log.
Correlated the error increase with release 2026.03.12.4.
Compared production against the previous deployment slot.
Confirmed the previous slot remained healthy.
Confirmed no matching Azure platform incident.

Agent finding:

HTTP 500 responses increased from 0.4% to 18% within three minutes of release 2026.03.12.4. Most failures reference a missing application setting. The previous deployment slot passes availability and dependency checks.

Notice the shape of that sentence: a delta, a time correlation, and a known-good comparison. That's what makes it actionable rather than merely true.

Agent action table

#	Action	Type	Detail
1	Investigate alert	Read	Reads HTTP 5xx rate, failed requests, latency, availability-test results
2	Analyze exceptions	Read	Identifies missing-configuration exception as dominant failure
3	Correlate deployment	Read	Finds release 2026.03.12.4 deployed three minutes before the spike
4	Compare slots	Read	Confirms previous slot healthy while production is failing
5	Rule out platform issue	Read	Checks Azure Resource Health and dependency health
6	Assess blast radius	Read	Determines checkout affected, unrelated services healthy
7	Create SNOW INC	Write (ITSM)	P1 against the Checkout CI, assigned to Application Operations
8	Update INC evidence	Write (ITSM)	Adds exceptions, deployment correlation, resource links, affected operations
9	Classify response	Decision	Classifies as deployment-caused; selects rollback
10	Pause release	Write (DevOps)	Pauses the failed release pipeline
11	Request approval	Decision	Requests approval to swap the previous healthy slot into production
12	Swap slot	Write (Azure)	Performs the approved swap only on the named App Service
13	Validate recovery	Validation	Confirms 5xx, latency, exceptions, dependencies, availability recover
14	Update/resolve INC	Write (ITSM)	Records approval, rollback, timestamps, recovery evidence
15	Create SNOW PRB	Write (ITSM)	Problem record for configuration-validation improvements

Recovery note

Production was reverted to the previous healthy deployment slot at 14:26 UTC. HTTP 500 responses declined from 18% to below 1%, and availability tests passed for 15 consecutive minutes. Preliminary cause: missing production configuration in release 2026.03.12.4.

Try it yourself

Break it:

# Put a healthy build in the 'previous' slot first, then break production # by deploying code that reads an app setting which only exists in staging. az webapp config appsettings delete -g $RG -n app-checkout-demo \ --setting-names Checkout__PaymentProviderKey az webapp restart -g $RG -n app-checkout-demo

Alert it:

az monitor metrics alert create -g $RG -n "alert-checkout-5xx" \ --scopes "/subscriptions/$SUB/resourceGroups/$RG/providers/Microsoft.Web/sites/app-checkout-demo" \ --condition "total Http5xx > 20" \ --window-size 5m --evaluation-frequency 1m --severity 1 \ --description "P1 – Checkout App Service returning elevated HTTP 500 responses"

Ask it (before the alert, to see the read phase in isolation):

app-checkout-demo is returning HTTP 500s. Do not change anything. Investigate and tell me: 1. the dominant exception and its share of total failures 2. what deployed in the 30 minutes before the error rate changed 3. whether the 'previous' slot is healthy right now 4. whether Azure Resource Health shows a platform issue Give me the evidence chain and the smallest reversible mitigation.

Expect it to propose:

Proposed action: Swap slot 'previous' into production on app-checkout-demo Risk: brief connection drain (~10s). Fully reversible by swapping back. Validation: Http5xx < 1% and availability test passing for 15 minutes. [Approve] [Deny]

Verify it:

requests | where timestamp > ago(1h) | summarize failed = countif(success == false), total = count() by bin(timestamp, 1m) | extend failureRate = 100.0 * failed / total | render timechart

🎬 Best thing to record for a demo: the moment the Approve button appears with the evidence already attached. That single screen is the whole value proposition.

Guardrails

Scope write permissions to the named App Service, not the resource group.
Verify the previous slot is healthy before swapping — a swap into a broken slot doubles the outage.
Require human approval for production slot swaps.
Preserve the failed deployment for analysis; don't let the pipeline overwrite it.
Never grant subscription-level Contributor or Owner.

Use case #2 — AKS: pods in CrashLoopBackOff

What happened

Image orders-api:4.18.0 referenced a configuration key that didn't exist in the production namespace. Eight of ten pods entered CrashLoopBackOff. The two survivors couldn't carry production traffic, producing latency and ingress 5xx.

How Azure Monitor detected it

Managed Prometheus / Container Insights detected:

Unavailable replicas
Increasing container restart count
CrashLoopBackOff pod state
Failed readiness and liveness checks
Elevated ingress HTTP 5xx
Reduced successful-request rate

P1 – Orders API unavailable replicas in production AKS

How the agent found the cause

Identified the cluster, namespace, and deployment.
Reviewed deployment availability and pod states.
Examined logs from failing containers.
Reviewed Kubernetes warning events.
Compared current and previous ReplicaSets.
Correlated the failure with revision 42.
Identified the missing configuration key.
Confirmed revision 41 was previously healthy.
Checked node CPU, memory, storage, networking, status.
Determined AKS infrastructure was healthy.

Agent finding:

Eight of ten Orders API pods entered CrashLoopBackOff during rollout of revision 42. Container logs show a missing configuration key. Revision 41, using image 4.17.6, was healthy.

Step 10 is the one people skip. "Rule out the infrastructure" is what stops you from spending an hour on a node pool that was never the problem.

Agent action table

#	Action	Type	Detail
1	Investigate workload	Read	Deployment status, unavailable replicas, readiness, restart counts
2	Analyze pod logs	Read	Startup failure from a missing configuration key
3	Analyze events	Read	Image pulls, mounts, scheduling, probes, container events
4	Correlate rollout	Read	Revision 42 deployed immediately before failure
5	Compare ReplicaSets	Read	Revision 41 was the previous healthy workload
6	Rule out infrastructure	Read	Nodes, memory, CPU, networking, storage healthy
7	Assess blast radius	Read	Eight of ten replicas unavailable
8	Create SNOW INC	Write (ITSM)	P1 against the Orders API CI
9	Update INC evidence	Write (ITSM)	Namespace, image, revision, errors, rollout correlation
10	Classify response	Decision	Application/configuration-caused
11	Request approval	Decision	Approval to pause and roll back revision 42
12	Pause failed rollout	Write (K8s/DevOps)	Prevents the release progressing or being reapplied
13	Roll back deployment	Write (K8s)	Rolls back only the named deployment to revision 41
14	Validate replicas	Validation	All expected replicas ready and stable
15	Validate application	Validation	Ingress errors decline; synthetic orders succeed
16	Update/resolve INC	Write (ITSM)	Revision, approval, rollback, recovery evidence
17	Create defect/task	Write (ITSM)	Add configuration validation to CI/CD

Recovery note

Orders API was rolled back from revision 42 to revision 41. All ten replicas are ready, restart counts have stabilized, ingress HTTP 5xx responses are below threshold, and synthetic order transactions are succeeding.

Try it yourself

Break it:

az aks get-credentials -g $RG -n aks-demo kubectl create namespace orders kubectl create deployment orders-api -n orders --image=nginx:1.25 --replicas=10 kubectl rollout status deployment/orders-api -n orders # revision 41 equivalent, healthy # Now break it: point at an image whose entrypoint requires a missing env var kubectl set image deployment/orders-api -n orders orders-api=busybox:1.36 kubectl patch deployment orders-api -n orders --type=json -p='[ {"op":"add","path":"/spec/template/spec/containers/0/command", "value":["sh","-c","test -n \"$ORDERS_CONFIG_KEY\" || (echo \"FATAL: missing ORDERS_CONFIG_KEY\" >&2; exit 1); sleep 3600"]} ]'

Watch it break:

kubectl get pods -n orders -w kubectl rollout history deployment/orders-api -n orders

Ask it:

Pods in namespace 'orders' on aks-demo are crash looping. Read-only. Tell me: which revision introduced it, the exact container error, whether the node pool is healthy, and how many replicas are actually serving. Then tell me the last known-good revision and why you believe it was healthy.

Expect it to propose: kubectl rollout undo deployment/orders-api -n orders --to-revision= scoped to that single deployment.

Verify it:

kubectl get deployment orders-api -n orders \ -o jsonpath='{.status.readyReplicas}/{.status.replicas}{"\n"}' kubectl get pods -n orders --no-headers | awk '{print $4}' | sort | uniq -c

Guardrails

Restrict access to the required cluster, namespace, and deployment.
Do not grant cluster-admin.
Require approval for rollback.
Coordinate with GitOps reconciliation. If Flux or Argo owns that deployment, a kubectl rollout undo gets reverted within minutes and you've created a flapping outage. Either pause reconciliation first or roll back through Git.
Do not permit namespace, persistent-volume, or cluster deletion.

Use case #3 — Azure SQL Database: CPU saturation and timeouts

The use case where the agent's job is to buy time, not to fix the problem.

What happened

A query execution plan changed after an application release. The new plan consumed substantially more CPU and workers. The database saturated, producing SQL dependency timeouts and failed customer requests.

How Azure Monitor detected it

Sustained CPU saturation
Worker or session pressure
Increased connection failures
SQL dependency timeouts
Query-duration deviation from baseline
Degraded customer API success rate

P1 – Customer API database saturation causing request timeouts

How the agent found the cause

Reviewed database CPU, workers, sessions, connections.
Reviewed application SQL dependency failures.
Checked for blocking and deadlocks.
Ran approved read-only Query Store diagnostics.
Identified the primary CPU-consuming query.
Detected a recent execution-plan change.
Correlated with an application release.
Ruled out Azure service health and storage issues.
Determined a temporary scale operation could restore service.
Left permanent query remediation to the DBA team.

Agent finding:

Database CPU has remained saturated for 17 minutes. One query accounts for most recent CPU consumption and changed execution plan shortly before the incident. Application SQL dependency timeout rate is 23%.

Step 10 is the design decision that makes this safe. The agent correctly diagnoses a plan regression and then deliberately does not fix it, because forcing a plan or dropping an index is a permanent, DBA-owned, change-controlled action. It buys capacity instead.

Agent action table

#	Action	Type	Detail
1	Investigate database	Read	CPU, workers, sessions, connections, storage, availability
2	Analyze app impact	Read	SQL dependency latency, failures, affected API operations
3	Check blocking	Read	Approved diagnostics for blocking, deadlocks, connection growth
4	Analyze Query Store	Read	Highest-impact query and recent plan change
5	Correlate changes	Read	Recent application and database deployments
6	Rule out platform issue	Read	Service health, storage, database availability
7	Assess blast radius	Read	Affected applications; other databases healthy
8	Create SNOW INC	Write (ITSM)	P1 against the production database CI
9	Update INC evidence	Write (ITSM)	Utilization, query ID, timeouts, change correlation
10	Classify response	Decision	Scaling is mitigation; query changes remain DBA-owned
11	Calculate bounded scale	Read	Selects the smallest pre-approved capacity increase
12	Request approval	Decision	Database owner or incident commander
13	Scale database	Write (Azure)	Increases the affected database by one approved service step
14	Validate recovery	Validation	CPU, workers, timeouts, application success recover
15	Update INC	Write (ITSM)	Previous/new capacity, approval, timing, results
16	Create SNOW PRB	Write (ITSM)	Permanent query-remediation work
17	Create scale-down task	Write (ITSM)	Task/change to restore normal capacity after stability

Recovery note

Azure SQL capacity was temporarily increased by one approved service step. CPU declined from sustained saturation to 54%, and SQL dependency timeouts returned to baseline. Query Store indicates a probable execution-plan regression requiring permanent DBA remediation.

Try it yourself

Break it — generate a saturating workload against db-customer (an unindexed LIKE '%...%' scan in a tight loop from a container in the same region works fine on an S1).

Alert it:

az monitor metrics alert create -g $RG -n "alert-sql-cpu" \ --scopes "/subscriptions/$SUB/resourceGroups/$RG/providers/Microsoft.Sql/servers/sqlsrv-sre-demo/databases/db-customer" \ --condition "avg cpu_percent > 90" \ --window-size 5m --evaluation-frequency 1m --severity 1 \ --description "P1 – Customer API database saturation causing request timeouts"

Ask it:

db-customer is saturated. Read-only investigation. Identify the top CPU-consuming query, whether its plan changed recently, and what application release correlates. Do NOT propose index, schema, plan-forcing, or session-kill actions. Propose only the smallest temporary capacity step that restores service, and tell me what it costs per day and when we should scale back down.

Read-only Query Store diagnostics the agent should run:

SELECT TOP 10 qsq.query_id, qsp.plan_id, qsp.last_execution_time, SUM(qsrs.count_executions) AS executions, SUM(qsrs.avg_cpu_time * qsrs.count_executions) AS total_cpu_us FROM sys.query_store_query AS qsq JOIN sys.query_store_plan AS qsp ON qsp.query_id = qsq.query_id JOIN sys.query_store_runtime_stats AS qsrs ON qsrs.plan_id = qsp.plan_id JOIN sys.query_store_runtime_stats_interval AS qsrsi ON qsrsi.runtime_stats_interval_id = qsrs.runtime_stats_interval_id WHERE qsrsi.start_time > DATEADD(hour, -2, GETUTCDATE()) GROUP BY qsq.query_id, qsp.plan_id, qsp.last_execution_time ORDER BY total_cpu_us DESC;

Expect it to propose: az sql db update -g $RG -s sqlsrv-sre-demo -n db-customer --service-objective S2 — exactly one step, on exactly that database.

Guardrails

Restrict scaling to the named database.
Define minimum and maximum capacity.
Require approval for scale-up and scale-down.
Time-limit temporary capacity — an un-reversed emergency scale-up is how a P1 becomes a budget incident.
Do not autonomously force plans, terminate sessions, modify indexes, or change schema.
Track the temporary cost impact with an Azure Cost Management alert.

Use case #4 — Azure Cosmos DB: HTTP 429 throttling

The one where the correct root cause is "we're succeeding."

What happened

A marketing campaign increased Product Catalog traffic by roughly 40%. The Cosmos DB container hit its provisioned throughput ceiling. HTTP 429s increased, and client retries amplified application latency.

How Azure Monitor detected it

High normalized RU consumption
Increased HTTP 429 responses
Elevated server-side latency
Application dependency failures
Sustained operation near the throughput limit

P2 – Cosmos DB throttling affecting Product Catalog requests

How the agent found the cause

Reviewed normalized RU consumption and throttled requests.
Identified the affected database and container.
Reviewed regional and partition behavior.
Checked for hot-partition evidence.
Reviewed application retry telemetry.
Compared current traffic with the historical baseline.
Correlated demand with the marketing campaign.
Checked recent application deployments.
Checked Azure service health.
Determined the primary cause was legitimate demand.

Agent finding:

The Product Catalog container is at its configured throughput ceiling, and 21% of requests are being throttled. Traffic increased by approximately 40% following a campaign launch. No deployment or regional platform issue correlates with the event.

Step 4 is the fork in the road. If consumption is uneven across partitions, more RU/s is money set on fire — the correct answer is an architecture change, not a scale-up. The agent has to check before it recommends.

Agent action table

#	Action	Type	Detail
1	Investigate throttling	Read	RU consumption, 429 rate, latency, requests, availability
2	Identify scope	Read	Account, database, container, operations, regions
3	Analyze demand	Read	Compares traffic and RU consumption with historical patterns
4	Check partitions	Read	Looks for uneven partition consumption where telemetry permits
5	Analyze retries	Read	Determines whether client retries are amplifying the incident
6	Correlate events	Read	Links demand to campaign traffic; excludes release/platform issues
7	Assess blast radius	Read	Affected catalog operations; unaffected containers
8	Create SNOW INC	Write (ITSM)	P2 against the Product Catalog CI
9	Update INC evidence	Write (ITSM)	RU, throttling, latency, traffic, partition evidence
10	Classify response	Decision	Demand-driven unless hot-partition evidence exists
11	Calculate throughput	Read	Smallest increase within the cost ceiling
12	Request approval	Decision	Approval for a temporary throughput increase
13	Increase throughput	Write (Azure)	Raises throughput only to the approved maximum
14	Validate recovery	Validation	429 rate and latency recover
15	Update/resolve INC	Write (ITSM)	Throughput, approval, cost implication, recovery
16	Create capacity task	Write (ITSM)	Work to return throughput to normal
17	Create SNOW PRB	Write (ITSM)	Partition or retry improvements, if required

Recovery note

Provisioned throughput was increased within the approved production limit. HTTP 429 responses declined from 21% to below 1%, and Product Catalog latency returned to baseline. The increase is temporary and will be reviewed after campaign traffic subsides.

Try it yourself

Break it: the container was created at 400 RU/s. Drive a few hundred reads per second at it and you'll be throttled within seconds.

Alert it:

az monitor metrics alert create -g $RG -n "alert-cosmos-429" \ --scopes "/subscriptions/$SUB/resourceGroups/$RG/providers/Microsoft.DocumentDB/databaseAccounts/cosmos-sre-demo" \ --condition "total TotalRequests where StatusCode == 429 > 100" \ --window-size 5m --evaluation-frequency 1m --severity 2 \ --description "P2 – Cosmos DB throttling affecting Product Catalog requests"

Ask it:

cosmos-sre-demo container 'products' is throttling. Read-only. Before recommending anything, tell me whether RU consumption is EVEN across physical partitions or concentrated. If it is concentrated, do not recommend a throughput increase — recommend an architecture problem record instead. If it is even, tell me the smallest RU/s that clears throttling and the daily cost delta.

That prompt is the whole use case. Getting the agent to refuse the easy answer under a stated condition is the skill.

Verify it:

AzureDiagnostics | where ResourceProvider == "MICROSOFT.DOCUMENTDB" | where Category == "DataPlaneRequests" | summarize throttled = countif(statusCode_s == "429"), total = count() by bin(TimeGenerated, 1m) | extend throttleRate = 100.0 * throttled / total | render timechart

Guardrails

Define a maximum throughput ceiling the agent may not exceed.
Restrict changes to the named container.
Require approval for increases and reductions.
Do not permit deletion, consistency-level changes, or region changes.
Create cost alerts for prolonged increased throughput.
Treat persistent hot partitions as an architecture issue, never as a scaling issue.

Use case #5 — Azure VM: OS/root disk full

The most operationally dangerous use case in this post, and the one with the most interesting guardrail design.

What happened

A legacy payment application generated excessive trace logs. Log rotation stopped working and the root filesystem filled. The VM stayed available at the Azure platform layer — heartbeat green, Resource Health fine — but the application stopped, because it could no longer write to disk.

This is the classic "green dashboard, dead service" failure. Platform-layer monitoring alone will never catch it.

How Azure Monitor detected it

Azure Monitor Agent and guest telemetry detected:

Critically low filesystem free space
Rapid filesystem consumption
Application process stopped
Failed availability tests
Elevated HTTP 5xx
Healthy VM heartbeat but unhealthy application

P1 – Legacy payment application unavailable due to full VM OS disk

How the agent found the cause

Confirmed the VM was online.
Confirmed Azure Monitor Agent heartbeat.
Identified the affected root/OS filesystem.
Reviewed the free-space trend.
Correlated application failure with disk exhaustion.
Found 86 GB of growth in the application trace directory.
Reviewed recent deployments and logging changes.
Checked log-rotation status.
Confirmed the files matched the approved cleanup policy.
Excluded customer data, database files, audit logs, and system files.

Agent finding:

The VM is healthy at the Azure platform layer, but the OS volume has less than 1% free space. The approved application trace directory grew by 86 GB in six hours. The payment service stopped when it could no longer write to disk.

Agent action table

#	Action	Type	Detail
1	Investigate VM	Read	VM, agent heartbeat, Azure platform health
2	Analyze filesystem	Read	Root volume and free-space trend
3	Correlate app failure	Read	Service stopped after disk exhaustion
4	Identify disk consumer	Read	Abnormal growth in an approved trace directory
5	Check changes	Read	Deployments, logging changes, rotation, scheduled tasks
6	Validate cleanup scope	Read	Candidate files meet approved path, type, and age rules
7	Protect data	Decision	Excludes databases, customer data, security logs, unknown files
8	Create SNOW INC	Write (ITSM)	P1 against the payment VM/application CI
9	Update INC evidence	Write (ITSM)	Disk usage, growth timeline, service impact, cleanup scope
10	Request approval	Decision	Approval for the restricted recovery runbook
11	Archive logs	Write (Guest OS)	Archives eligible files to protected Azure Storage
12	Remove eligible files	Write (Guest OS)	Removes only successfully archived, allowlisted files
13	Run log rotation	Write (Guest OS)	Executes the approved rotation operation
14	Restart service	Write (Guest OS)	Restarts only the named payment service if necessary
15	Validate recovery	Validation	Disk, application, archive, availability, growth stabilization
16	Update/resolve INC	Write (ITSM)	Bytes processed, exclusions, approval, results
17	Create SNOW PRB	Write (ITSM)	Permanent logging and rotation remediation

Recovery note

The approved recovery runbook archived and removed 82 GB of eligible application trace files. OS-volume free space is now 31%. The payment service was restarted and has passed health checks for 15 minutes. No database, customer, security, or system files were modified.

That last sentence is not decoration. It is the sentence your auditor will read.

Try it yourself

Break it (lab VM only — this fills the root disk):

az vm run-command invoke -g $RG -n vm-payments-linux \ --command-id RunShellScript --scripts " mkdir -p /var/log/payments/trace fallocate -l 24G /var/log/payments/trace/trace-$(date +%s).log df -h / "

Ask it:

vm-payments-linux root filesystem is nearly full and the payment service is down. Read-only first. Tell me: - exactly which directory grew, by how much, over what window - whether log rotation is configured and when it last ran - what changed in the last 24 hours that would explain it Then tell me which files are inside the approved cleanup allowlist (/var/log/payments/trace/*.log, older than 2h) and which are NOT, and confirm no database, audit, or customer data files are in scope. Do not delete anything.

The critical design point. SRE Agent blocks delete and remove commands outright. You cannot have it rm those files through its Azure CLI surface, and you should be glad. The correct implementation is a fixed-purpose, version-controlled runbook that the agent invokes with tightly bounded parameters:

# What the agent is allowed to call — one runbook, allowlisted parameters, nothing else az automation runbook start \ -g $RG --automation-account-name aa-sre-runbooks \ -n "Reclaim-TraceDiskSpace" \ --parameters vmName=vm-payments-linux \ allowedPath=/var/log/payments/trace \ pattern='*.log' \ minAgeHours=2 \ maxBytes=90000000000 \ archiveToContainer=payments-trace-archive \ requireArchiveBeforeDelete=true \ dryRun=false

The runbook — not the agent — owns the destructive logic, and it enforces:

path allowlist (refuse anything outside allowedPath)
filename pattern allowlist
minimum file age
maximum total bytes per execution
successful archive verified before any deletion
hard-refuse if any candidate file is unknown, or matches a protected pattern (*.mdf, *.bak, /var/log/audit/*, *.key, *.pem)
a cooldown that prevents re-execution within N hours

Verify it:

az vm run-command invoke -g $RG -n vm-payments-linux \ --command-id RunShellScript \ --scripts "df -h /; systemctl is-active payments.service; ls -la /var/log/payments/trace | head"

Guardrails

Do not give SRE Agent unrestricted SSH or shell access. Ever. This is the single highest-leverage rule in this post.
Use a version-controlled, fixed-purpose runbook.
Allowlist paths, patterns, file ages, and maximum cleanup size.
Stop if the responsible files are unknown. A disk filled by something you can't identify is a security event until proven otherwise.
Require successful archival before deletion.
Prevent repeated execution with a cooldown.
Treat cleanup as temporary mitigation — the problem record is the fix.

Use case #6 — Linux VM: anomalous CPU saturation

Where "anomalous" is doing all the work.

What happened

Release 7.3.1 introduced an immediate retry loop when an inventory dependency failed. The application retried continuously without backoff, consuming nearly all VM CPU and causing request timeouts.

How Azure Monitor detected it

The alert deliberately combined conditions rather than firing on a threshold:

CPU significantly outside the historical baseline
Persistent saturation over multiple evaluations
Increased request latency
Increased dependency failures
No approved maintenance or batch workload active

P1 – Production order-processing Linux VM experiencing anomalous CPU saturation

A static "CPU > 90%" rule on a batch-processing VM is a pager that everyone learns to ignore. The composite condition is what makes the alert worth waking someone — or an agent — for.

How the agent found the cause

Confirmed the VM was online.
Compared current CPU with historical behavior.
Identified the order-processing service as the main CPU consumer.
Reviewed application request latency.
Reviewed downstream dependency failures.
Analyzed application retry logs.
Correlated CPU growth with release 7.3.1.
Excluded expected batch jobs.
Excluded Azure maintenance or platform issues.
Confirmed other application instances had sufficient capacity.

Agent finding:

CPU increased from a normal range of 35–50% to 98% four minutes after deployment 7.3.1. The order-processing service is repeatedly calling a failed dependency without backoff. No expected batch job or platform maintenance is active.

Agent action table

#	Action	Type	Detail
1	Confirm anomaly	Read	Compares current CPU with the historical baseline
2	Analyze impact	Read	Latency, timeouts, availability, dependencies
3	Identify process	Read	Named order service is the primary CPU consumer
4	Correlate deployment	Read	Release 7.3.1, four minutes before saturation
5	Analyze logs	Read	Dependency retry loop without backoff
6	Rule out expected work	Read	Excludes batch, backup, maintenance, scheduled processing
7	Assess blast radius	Read	Traffic and available capacity on other instances
8	Create SNOW INC	Write (ITSM)	P1 against the Order Processing CI
9	Update INC evidence	Write (ITSM)	CPU, process, dependency, release, customer impact
10	Classify response	Decision	Rollback, not VM resize or arbitrary process termination
11	Request approval	Decision	Approval to drain, roll back, restart the service
12	Drain VM	Write (Azure)	Removes the VM from load-balancer rotation
13	Restore prior release	Write (Guest OS/DevOps)	Restores known-good version or configuration
14	Restart named service	Write (Guest OS)	Restarts only `orders-service`
15	Validate recovery	Validation	CPU, retries, latency, dependencies, health recover
16	Return to rotation	Write (Azure)	Restores traffic only after successful validation
17	Update/resolve INC	Write (ITSM)	Drain, rollback, restart, approval, results
18	Create defect	Write (ITSM)	Retry, backoff, and circuit-breaker remediation

Steps 12 and 16 are the pattern to steal: drain before you touch, restore traffic only after validation passes. Most homegrown automation restarts a service while it's still taking traffic and turns a degradation into an outage.

Recovery note

Release 7.3.1 introduced a retry loop when the inventory dependency failed. After approval, the VM was drained, the previous release was restored, and orders-service was restarted. CPU declined from 98% to 43%, and application latency returned to baseline.

Try it yourself

Break it:

az vm run-command invoke -g $RG -n vm-payments-linux \ --command-id RunShellScript --scripts " nohup bash -c 'while true; do curl -s -m 1 http://127.0.0.1:9/inventory >/dev/null 2>&1; done' & nohup bash -c 'while true; do :; done' & echo started "

Ask it:

CPU on vm-payments-linux is at 98%. Read-only. First: is this actually anomalous, or is it consistent with this VM's historical pattern for this hour and day of week? Show me the baseline. If anomalous: which process, which dependency is it calling, at what rate, and what deployed immediately before? Do NOT propose killing the top process or resizing the VM.

Expect it to propose a drain → restore → restart sequence, with the drain step as a separate approval from the restart.

Guardrails

Do not automatically terminate the highest-CPU process. It is very often a legitimate workload, and occasionally it's a security incident you just destroyed the evidence for.
Allow operations only for named services.
Drain before service restart where possible.
Escalate unknown or suspicious processes to security rather than remediating them.
Limit restart attempts.
Do not permanently resize the VM when the evidence points to faulty code. Resizing to survive a retry storm is buying hardware to host a bug.

Use case #7 — Windows VM with IIS: memory leak

What happened

Release ClaimsPortal 5.9.0 introduced a memory leak in the Claims IIS application pool. Memory consumption climbed over several hours, paging began, request queues grew, and IIS returned HTTP 503.

How Azure Monitor detected it

Azure Monitor Agent collected available memory, committed bytes, paging activity, process working set, IIS request queues, HTTP 500/503 responses, and availability-test results. The alert required sustained abnormal growth, not a brief spike.

P1 – Memory exhaustion affecting production IIS application

How the agent found the cause

Confirmed the VM and monitoring agent were healthy.
Compared memory behavior with the historical baseline.
Identified the relevant w3wp.exe process.
Mapped it to the Claims application pool.
Reviewed paging and request queues.
Correlated HTTP 503 errors with low available memory.
Reviewed Windows Event Logs.
Correlated the growth with release 5.9.0.
Excluded antivirus and scheduled-reporting activity.
Confirmed another instance could carry traffic.

Agent finding:

Available memory declined from 42% to 4% over three hours. The Claims application pool grew from 1.8 GB to 11.6 GB without releasing memory after traffic normalized. Paging and HTTP 503 errors followed. The pattern began after release 5.9.0.

"Without releasing memory after traffic normalized" is the sentence that distinguishes a leak from load. Cache growth under load is normal; failure to return afterwards is not.

Agent action table

#	Action	Type	Detail
1	Confirm anomaly	Read	Memory, committed bytes, paging vs. baseline
2	Identify process	Read	Maps growing `w3wp.exe` to the Claims pool
3	Analyze IIS health	Read	Pools, queues, HTTP errors, availability
4	Correlate release	Read	Links sustained growth to release 5.9.0
5	Rule out other causes	Read	Excludes scheduled jobs, antivirus, maintenance
6	Assess capacity	Read	Confirms another instance can serve traffic
7	Create SNOW INC	Write (ITSM)	P1 against the Claims Portal CI
8	Update INC evidence	Write (ITSM)	Memory, paging, pool, release, HTTP impact
9	Classify response	Decision	Targeted recycling, not a full VM restart
10	Request approval	Decision	Approval to drain and recycle the named pool
11	Drain VM	Write (Azure)	Removes VM from load-balancer rotation
12	Capture diagnostics	Write (Guest OS)	Captures approved diagnostics to a protected location
13	Recycle app pool	Write (Guest OS)	Recycles only the Claims application pool
14	Validate recovery	Validation	Memory, paging, queues, HTTP errors, availability recover
15	Return to rotation	Write (Azure)	Restores traffic after health checks pass
16	Update/resolve INC	Write (ITSM)	Memory before/after, approval, stability
17	Create SNOW PRB	Write (ITSM)	Memory-leak remediation

Step 12 before step 13 matters: recycling the pool destroys the evidence. Capture first.

Recovery note

Abnormal memory growth was isolated to the Claims application pool following release 5.9.0. The VM was drained, the approved application pool was recycled, and health checks passed before traffic was restored. Available memory increased from 4% to 61%, and HTTP 503 responses stopped.

Try it yourself

Break it (lab VM only):

az vm run-command invoke -g $RG -n vm-claims-win ` --command-id RunPowerShellScript --scripts " Install-WindowsFeature Web-Server -IncludeManagementTools New-WebAppPool -Name 'ClaimsPool' # Simulate the leak \$leak = New-Object System.Collections.ArrayList 1..40 | ForEach-Object { [void]\$leak.Add((New-Object byte[] 100MB)) ; Start-Sleep -Milliseconds 200 } "

Ask it:

vm-claims-win available memory is at 4%. Read-only. Map the growing process to an IIS application pool. Show me the memory curve for the last 6 hours and tell me whether memory was released after traffic dropped. Correlate with deployment history. Then propose the most targeted possible mitigation — I do not want a VM restart.

Expect it to propose:

Restart-WebAppPool -Name "ClaimsPool"

…and nothing else on that machine.

Guardrails

Recycle only the named application pool.
Avoid full VM restart as the first response — it's a bigger hammer with a longer outage and it destroys the leak evidence.
Do not copy memory dumps into ServiceNow. They contain credentials, tokens, and customer data.
Store diagnostics in a secured location; put the link in the ticket.
Prevent repeated automatic recycling — a pool that needs recycling every 40 minutes is an incident, not a routine.
Escalate if the leak returns during the observation period.

Use case #8 — Virtual Machine Scale Set: unhealthy instance

The most autonomy-ready use case in the list, and the reason is stateless workloads.

What happened

A configuration extension failed while VMSS instance 17 was being provisioned. The VM was running but its application service never started. The instance failed application health probes and caused intermittent errors.

How Azure Monitor detected it

Reduced healthy backend count
Application Health extension failure
Backend health-probe failure
VM extension provisioning failure
Instance-specific errors
Elevated intermittent HTTP 5xx

P2 – Unhealthy VM Scale Set instance causing intermittent API failures

How the agent found the cause

Reviewed the VMSS healthy-instance count.
Identified instance 17 as the only unhealthy instance.
Reviewed backend health.
Compared instance 17 with healthy instances.
Checked the image and VMSS model.
Reviewed VM extension state.
Found the configuration extension failure.
Reviewed boot and application diagnostics.
Confirmed the workload was stateless.
Confirmed eleven instances could carry production traffic.

Agent finding:

Instance 17 is the only unhealthy member of the 12-instance scale set. Its application health probe has failed since 09:18 UTC. The configuration extension failed during provisioning, and the application service never started. Eleven healthy instances can maintain service.

Agent action table

#	Action	Type	Detail
1	Investigate VMSS	Read	Instance health, provisioning state, healthy count
2	Identify instance	Read	Instance 17 is the only unhealthy member
3	Analyze backend health	Read	Confirms the instance fails application probes
4	Compare instances	Read	Image, model, extensions, configuration
5	Find extension failure	Read	Locates the failed configuration extension
6	Review diagnostics	Read	Boot, extension, and application diagnostics
7	Assess safe capacity	Read	Eleven instances can carry traffic
8	Confirm statelessness	Decision	Replacement won't destroy required local state
9	Create SNOW INC	Write (ITSM)	P2 against the VMSS/application CI
10	Update INC evidence	Write (ITSM)	Instance, extension error, health, capacity evidence
11	Classify response	Decision	Replacement or reimage per approved procedure
12	Request approval	Decision	Approval to isolate and replace instance 17
13	Isolate instance	Write (Azure)	Ensures the instance receives no production traffic
14	Preserve evidence	Read/Write	Stores approved diagnostic evidence securely
15	Replace instance	Write (Azure)	Reimages or replaces only instance 17
16	Validate provisioning	Validation	Image, model, and extensions deploy successfully
17	Validate service	Validation	Backend health, capacity, customer errors recover
18	Update/resolve INC	Write (ITSM)	Replacement, approval, diagnostics, recovery
19	Create SNOW PRB	Write (ITSM)	Extension and image-validation improvement work

Steps 8 and 7 are the gate. Replacing a stateless instance with eleven healthy peers is genuinely low risk. Replacing a stateful instance, or replacing one when you're already at minimum capacity, is an outage. Both must be confirmed, not assumed.

Recovery note

VMSS instance 17 was isolated after its configuration extension failed and the application service did not start. Diagnostic evidence was captured, and the instance was replaced after approval. The replacement passed extension, application, and backend health checks. The scale set has returned to 12 healthy instances.

Try it yourself

Break it:

INSTANCE_ID=$(az vmss list-instances -g $RG -n vmss-api-demo \ --query "[5].instanceId" -o tsv) az vmss extension set -g $RG --vmss-name vmss-api-demo \ --name CustomScript --publisher Microsoft.Azure.Extensions \ --settings '{"commandToExecute":"exit 1"}'

Ask it:

vmss-api-demo has an unhealthy instance. Read-only. Identify which instance, since when, and the specific extension error. Confirm for me: (a) the workload is stateless, (b) how many healthy instances remain, and (c) whether remaining capacity can carry current traffic with 20% headroom. Only if all three are satisfied, propose a reimage of that single instance. Capture diagnostics before proposing anything.

Expect it to propose:

az vmss reimage -g $RG -n vmss-api-demo --instance-id $INSTANCE_ID

Verify it:

az vmss get-instance-view -g $RG -n vmss-api-demo --instance-id $INSTANCE_ID \ --query "vmHealth.status.code" az vmss list-instances -g $RG -n vmss-api-demo -o table

Guardrails

Confirm the workload is stateless before any replacement.
Confirm sufficient healthy capacity first.
Preserve diagnostic evidence before replacement — the instance is your only copy of the failure.
Restrict permissions to the named VMSS.
Limit simultaneous replacements. One at a time. An agent that reimages six instances because six probes failed has just caused the outage it was investigating.
Do not permit deletion of the entire scale set.

Use case #9 — Application Gateway: HTTP 502 from unhealthy backends

The cross-component change nobody coordinated.

What happened

Customer Portal release 9.2 changed the backend listener from port 443 to 8443. Application Gateway remained configured to connect on 443. All backend probes failed and customers received HTTP 502.

How Azure Monitor detected it

Increased Application Gateway HTTP 502 responses
Increased failed requests
Four unhealthy backends
Reduced healthy-host count
Failed synthetic availability tests

P1 – Application Gateway returning HTTP 502 due to unhealthy backends

How the agent found the cause

Reviewed Application Gateway metrics.
Retrieved backend health.
Identified the affected pool.
Reviewed probe path, protocol, host header, and port.
Reviewed backend settings.
Confirmed the application responded on 8443.
Confirmed the gateway used 443.
Correlated the mismatch with release 9.2.
Reviewed NSG, route, DNS, certificate, and WAF changes.
Excluded networking, certificate, and platform-health issues.

Agent finding:

HTTP 502 responses began at 16:07 UTC. All four Customer Portal backends are unhealthy. Release 9.2 changed the backend listener to port 8443, while Application Gateway continues to use port 443. No NSG, routing, or certificate issue correlates with the incident.

Step 9 is what separates a real investigation from a lucky guess. A 502 has at least six plausible causes — NSG, UDR, DNS, expired cert, WAF rule, backend down. The agent has to exclude them, in writing, before you trust the conclusion.

Agent action table

#	Action	Type	Detail
1	Investigate gateway	Read	HTTP 502, failed requests, latency, backend counts
2	Inspect backend health	Read	Identifies four unhealthy Customer Portal backends
3	Review configuration	Read	Settings, probes, protocol, port, TLS, routing
4	Test backend state	Read	App responds on 8443 while gateway uses 443
5	Correlate changes	Read	Links mismatch to Customer Portal release 9.2
6	Rule out networking	Read	NSGs, routes, DNS, TLS, WAF, platform health
7	Assess blast radius	Read	Confirms the Customer Portal pool is unavailable
8	Create SNOW INC	Write (ITSM)	P1 against the portal/gateway CI
9	Update INC evidence	Write (ITSM)	502s, backend, port, deployment, known-good configuration
10	Classify response	Decision	Restore the known-good backend listener
11	Request approval	Decision	Approval to restore source-controlled configuration
12	Restore listener	Write (Application)	Restores the application listener to approved port 443
13	Validate backend health	Validation	All four backends become healthy
14	Validate application	Validation	HTTP 502 declines; synthetic transactions pass
15	Verify controls	Read	Confirms no WAF, TLS, routing, or NSG control was weakened
16	Update/resolve INC	Write (ITSM)	Cause, restoration, approval, recovery
17	Create SNOW CHG	Write (ITSM)	Coordinated change for the intended port migration
18	Create SNOW PRB	Write (ITSM)	Cross-component deployment-validation work

Step 12 is a genuinely important choice. There were two ways to fix this: change the application back to 443, or change the gateway to 8443. The agent restores the application to the known-good, source-controlled state rather than mutating the gateway to match an unapproved change. One of those is a rollback; the other is ratifying an unreviewed change during an outage. Then step 17 files a proper change record for the migration the team clearly intended to do.

Step 15 exists because the fastest way to make a 502 disappear is to disable TLS validation. The agent must prove it didn't take the fast way.

Recovery note

Customer Portal release 9.2 changed the backend listener from port 443 to 8443 without a coordinated gateway change. The application listener was restored to the previous configuration. All four backends are healthy, HTTP 502 responses returned to baseline, and synthetic login tests passed for 15 minutes.

Try it yourself

Ask it:

appgw-portal is returning 502s and all backends are unhealthy. Read-only. Walk me through the exclusion, explicitly, for each of: NSG, UDR/route table, DNS resolution, backend TLS certificate, WAF rule blocking, backend process down, and probe configuration mismatch. State which you ruled out and the evidence for each. Then tell me the known-good configuration and where it is source-controlled.

Verify it:

az network application-gateway show-backend-health \ -g $RG -n appgw-portal \ --query "backendAddressPools[].backendHttpSettingsCollection[].servers[].{addr:address,health:health}" -o table

Guardrails

Do not disable TLS validation. Not to test, not temporarily, not "just to confirm."
Do not weaken NSGs or WAF policies as a mitigation.
Use source-controlled configuration as the definition of "known-good."
Restore a known-good state during the incident; migrate through a change record afterwards.
Require approval for gateway or backend changes.
Validate full application transactions, not just health probes. A probe returning 200 on /health proves very little.

Use case #10 — Azure Service Bus: queue and dead-letter backlog

The one that teaches the most important lesson in the entire post.

What happened

Release fulfillment-worker:6.4.0 couldn't deserialize messages containing a new deliveryWindow field. Consumer throughput dropped by 92%. The active backlog grew rapidly, and incompatible messages entered the dead-letter queue.

How Azure Monitor detected it

Increasing active-message count
Increasing oldest-message age
Dead-letter growth
Reduced completed-message rate
Consumer application errors
Delayed downstream business processing

P1 – Production order-event backlog delaying fulfillment

How the agent found the cause

Reviewed active, incoming, outgoing, and dead-letter counts.
Calculated message arrival and completion rates.
Confirmed the backlog was growing.
Reviewed consumer instance health.
Reviewed consumer errors and restarts.
Checked downstream dependency health.
Checked Service Bus authentication and authorization.
Correlated the throughput decline with release 6.4.0.
Found deserialization errors for the new field.
Determined scaling more broken consumers would amplify failures.

Agent finding:

The order-events queue grew from 3,000 to 185,000 active messages in 35 minutes. Consumer throughput dropped by 92% immediately after release fulfillment-worker:6.4.0. Application logs show deserialization failures involving the new deliveryWindow field.

Step 10 is the whole reason to use a reasoning agent instead of an autoscale rule. Every metric here screams "scale out the consumers." An HPA would have done exactly that, and every new replica would have dead-lettered messages faster.

Agent action table

#	Action	Type	Detail
1	Investigate queue	Read	Active, incoming, outgoing, scheduled, DLQ counts
2	Calculate flow rates	Read	Confirms arrival exceeds completion; backlog growing
3	Inspect consumers	Read	Health, instance count, scaling, errors, dependencies
4	Analyze failures	Read	Deserialization exceptions involving the new field
5	Correlate release	Read	Links the 92% throughput reduction to release 6.4.0
6	Rule out Service Bus	Read	Health, authorization, throttling, networking, service status
7	Assess blast radius	Read	Backlog age and fulfillment impact
8	Protect message data	Decision	Prevents payloads or personal data entering ServiceNow
9	Create SNOW INC	Write (ITSM)	P1 against the fulfillment integration CI
10	Update INC evidence	Write (ITSM)	Backlog, age, flow, exception, release, business impact
11	Classify response	Decision	Rollback, not scaling broken consumers
12	Request approval	Decision	Approval to restore consumer 6.3.7
13	Roll back consumer	Write (Azure/K8s)	Rolls back through the approved deployment platform
14	Restore capacity	Write (Azure/K8s)	Restores approved consumer instance count
15	Validate processing	Validation	Consumer and downstream health
16	Validate backlog	Validation	Completion exceeds arrival; DLQ growth stops
17	Update INC	Write (ITSM)	Rollback, rates, estimated drain time, approval
18	Maintain incident	Decision	Keeps the incident open until backlog age meets the objective
19	Create SNOW CHG	Write (ITSM)	Separate controlled change for DLQ replay
20	Create SNOW PRB	Write (ITSM)	Message-contract compatibility remediation

Recovery note

Consumer release 6.4.0 could not deserialize messages containing the new deliveryWindow field. The fulfillment worker was rolled back to 6.3.7. Consumer throughput recovered, new dead-letter growth stopped, and the active backlog is draining at approximately 7,500 messages per minute. Dead-letter replay requires a separately approved procedure.

The lesson: technical recovery is not business recovery

Step 18 is the most important row in this entire post.

At the moment of rollback, every technical signal is green. Consumers are healthy. Throughput has recovered. The DLQ has stopped growing. An agent optimizing for metrics would resolve the incident right there and go back to sleep.

But there are still 185,000 unshipped orders and a dead-letter queue full of messages that need a separately approved replay procedure. Customers are still affected. The incident stays open until backlog age meets the business objective, not until the graphs look nice.

Encode this in the response plan explicitly:

Do not resolve this incident when consumer health recovers. Resolution criteria: 1. Completion rate exceeds arrival rate for 15 consecutive minutes, AND 2. Oldest active message age is under 5 minutes, AND 3. Dead-letter count has not increased for 30 minutes. DLQ replay is out of scope for this incident. File a separate change record.

Try it yourself

Break it:

# Flood the queue while no consumer is running for i in $(seq 1 5000); do az servicebus queue message send -g $RG --namespace-name sb-sre-demo \ -q order-events --body "{\"orderId\":$i,\"deliveryWindow\":\"2026-08-09T10:00Z\"}" 2>/dev/null done az servicebus queue show -g $RG --namespace-name sb-sre-demo -n order-events \ --query "countDetails" -o json

Ask it:

order-events on sb-sre-demo has a growing backlog. Read-only. Give me: arrival rate, completion rate, current active count, oldest message age, DLQ count and DLQ growth rate, and the projected drain time at current rates. Then tell me why scaling out consumers is or is not the correct action here. Do not include any message payloads or customer data in your answer.

That last line is not optional. Message bodies routinely contain names, addresses, and payment references — and everything the agent writes goes into a ticket.

Verify it:

az servicebus queue show -g $RG --namespace-name sb-sre-demo -n order-events \ --query "{active:countDetails.activeMessageCount, dlq:countDetails.deadLetterMessageCount}" -o json

Guardrails

Never automatically purge queues.
Do not automatically replay dead-letter messages. Replay without idempotency guarantees means duplicate charges and duplicate shipments.
Do not place message payloads in ServiceNow.
Confirm idempotency before replay.
Scale consumers only when the processing path is healthy.
Require separate approval for replay or queue configuration changes.
Keep the incident open until business recovery is confirmed.

7. The ITSM integration model

Recommended incident fields

Field	Example
Short description	Production Orders API pods failing after deployment
Configuration item	`prod-aks-orders-api`
Assignment group	Container Platform Operations
Impact	High
Urgency	High
Environment	Production
Azure resource ID	Full affected Azure resource ID
Azure alert ID	Azure Monitor alert correlation identifier
SRE investigation	Link to the SRE Agent investigation thread
Current impact	Eight of ten replicas unavailable
Probable cause	Missing configuration in latest release
Confidence	High
Proposed action	Roll back to revision 41
Approval	Approver and UTC timestamp
Validation	Ten replicas ready and synthetic tests passing
Resolution	Service restored through rollback

The Confidence field earns its place. An agent that says "probable cause: missing configuration (confidence: low)" is far more useful than one that always sounds certain, because it tells the human how much to verify before approving.

Fields the agent can set directly (preview): assignment_group, category, subcategory, impact, urgency, priority, short_description, and any custom u_* field. It cannot change incident state through field updates — acknowledge and resolve are separate, dedicated tools.

Recommended agent-generated timeline

Every work note should carry:

Element	Example
Timestamp	`2026-03-18 14:26 UTC`
Observation	HTTP 500 rate increased to 18%
Evidence	Link to the Application Insights query
Change correlation	Incident started three minutes after deployment
Probable cause	Missing production application setting
Confidence	High
Proposed action	Swap to previous healthy slot
Risk	Temporary deployment rollback
Approval	Incident commander and timestamp
Execution	Slot-swap operation and result
Validation	5xx below 1% for 15 minutes
Follow-up	Problem record for configuration validation

Note that Evidence is a link, not a paste. This is a deliberate data-protection pattern: the ticket carries a pointer to the query, and the query results stay in the system that already has the right access controls.

Deduplication strategy

Do not create one incident per alert, pod, queue, or VMSS instance. Correlate on:

application/business service + Azure resource ID + environment (production) + alert-rule family + active incident time window

Related alerts attach to the existing incident as evidence or child alerts. Azure Monitor already merges recurring alerts into a single thread when it's the bound platform; for ServiceNow, this correlation key is yours to implement.

Get this wrong and your first AKS incident produces eight incidents, eight investigations, and eight rollback proposals for the same deployment.

Record responsibilities

Record	Purpose	Example
Incident (INC)	Restore service quickly and safely	App Service HTTP 500 outage
Problem (PRB)	Identify and remove the underlying cause	Missing deployment configuration validation
Change (CHG)	Govern permanent or higher-risk production changes	Coordinated Application Gateway port migration
Engineering defect/task	Correct application or automation behavior	Add retry backoff to the Linux application

The agent must clearly distinguish temporary mitigation from permanent correction. Every single use case above ends with a follow-up record, and that's not bureaucratic theatre — an agent that mitigates flawlessly and never files a problem record is an agent that lets the same outage recur forever while making the metrics look great.

8. Reality check: where you have to build

This is the section that will save you a month.

The PDF this post is built from is explicit that these are target response patterns, not guaranteed zero-configuration behavior. Having now checked each pattern against the product documentation, here is exactly where the gaps are and what fills them.

#	The pattern assumes	What's actually documented	What you must build
1	Agent creates a ServiceNow INC	ServiceNow is an inbound platform. Documented writes: post discussion entries, acknowledge, resolve, plus field updates (preview)	Incidents should originate in ServiceNow (via its own Azure Monitor integration) and flow in. If you truly need agent-initiated creation, add a Python tool or MCP server against the ServiceNow Table API
2	Agent creates PRB / CHG / defect records	Not a documented first-class ServiceNow action	Same: a custom tool against `/api/now/table/problem` and `/change_request`. This is ~30 lines of Python and worth doing properly, with a least-privileged integration user
3	ServiceNow and PagerDuty both connected	Only one incident platform active at a time; switching disconnects the other	Bind the agent to your system of record (ServiceNow). Reach the pager through a connector, Teams/Slack, or a webhook
4	Agent runs guest-OS cleanup (`rm`, rotate, restart)	`delete` and `remove` commands are blocked outright. `az keyvault` blocked. Management locks respected	Wrap all guest-OS work in fixed-purpose Azure Automation runbooks or a constrained `az vm run-command` script, invoked with allowlisted parameters. See use case #5
5	Agent "pauses the release pipeline"	Requires the GitHub or Azure DevOps connector, plus permissions on that pipeline	Connect source control; grant pipeline permissions explicitly; test the pause path before you need it
6	Approval gate on every action	Review mode shows Approve/Deny only for Azure infrastructure operations. Emails, Teams posts, and external queries proceed on the agent's reasoning	Use hooks or tool access policies to gate non-Azure actions
7	Agent resolves incidents	Supported — but during a pilot you don't want it	Require human confirmation before resolve. Encode it in the response plan and enforce it with a `Stop` hook
8	One agent handles everything	Response plans route to custom agents; skills cap at five concurrent active	Build domain specialists (§4.6). A single mega-agent thrashes its skill budget
9	Autonomous mode by default is fine	Per-plan default is Autonomous, and connecting a platform auto-creates an autonomous `quickstart_handler`	Delete the quickstart plan. Set every plan to Review explicitly
10	Agent has broad subscription rights	You can't remove individual permissions — only whole resource groups	Design resource groups as blast-radius boundaries before onboarding

None of these are blockers. All of them are a week of work you'd rather discover now than during your pilot readout.

9. Approval and autonomy policy

The policy I would actually ship:

Action category	Recommended initial policy
Read metrics, logs, traces, resource health	Automatic
Correlate deployments and configuration changes	Automatic
Create a ServiceNow incident	Automatic after deduplication
Add sanitized work notes	Automatic
Prepare a remediation plan	Automatic
Create a draft ServiceNow change	Automatic, but not approve it
Modify an Azure production resource	Human approval required
Execute a VM guest runbook	Human approval required
Roll back an application	Human approval required
Delete or replace a stateless VMSS instance	Human approval required
Resolve a ServiceNow incident	Human confirmation during the pilot
Delete data, purge queues, replay DLQ messages	Separate explicit approval
Autonomous remediation	Only after a proven, bounded pilot

Two notes on making this real:

Start in Review and stay there longer than feels necessary. The documented recommendation is to observe for two to four weeks and then promote specific triggers you consistently approve. Not the agent — the triggers. Promotion should be per-response-plan and evidence-based: "we approved this exact rollback proposal eleven times without modification" is a reason to go autonomous. "It seems good" is not.

Autonomy should be earned per action type, not per environment. "Autonomous in staging" is a fine starting rule, but the durable version is "autonomous for VMSS single-instance reimage where the workload is stateless and healthy capacity exceeds 80%" — a narrow, well-characterized action with a mechanical precondition.

10. Cross-cutting security controls

Use a dedicated managed identity for SRE Agent.
Scope Azure roles to selected resources or resource groups.
Avoid broad Contributor and Owner assignments.
Use fixed-purpose Automation runbooks for guest operations.
Do not provide unrestricted SSH, shell, or PowerShell execution.
Use a dedicated least-privileged ServiceNow integration identity.
Prefer OAuth or a managed connector over stored credentials.
Store required secrets in Key Vault — never in prompts. (The agent blocks az keyvault commands entirely, which helps.)
Sanitize logs before posting them into ServiceNow.
Do not post tokens, personal data, SQL text, message payloads, or memory dumps.
Record every approval and production action.
Define remediation cooldowns and maximum retry counts.
Require post-action application validation.
Retain existing manual runbooks as fallback.
Use Azure Cost Management alerts for temporary scaling actions.

What the platform gives you for free

Worth knowing so you don't rebuild it:

Layer	Isolation model
Compute	Dedicated sandbox (micro VM) per agent; tool execution separate from the reasoning loop
Database	Separate database per agent
Blob storage	Separate blob storage per agent
Network	Per-agent proxy instance validating every outbound request
Credentials	Identity sidecar issues short-lived, per-call tokens; credentials never enter the reasoning context

Token lifetimes: managed identity ~1 hour (auto-refreshed), OAuth refreshed 20 minutes before expiry, per-tool-call action tokens are single-use, blob SAS 1 hour refreshed at 45 minutes. Each tool invocation launches a fresh process whose entire tree terminates on completion — there are no persistent process pools, so one tool call cannot see another's environment.

The audit trail

Every az command is logged to your Application Insights as an AgentAzCliExecution custom event. This is your evidence for change management:

customEvents | where name == "AgentAzCliExecution" | where timestamp > ago(30d) | project timestamp, command = tostring(customDimensions.command), resource = tostring(customDimensions.resourceId), succeeded = tostring(customDimensions.success), thread = tostring(customDimensions.threadId) | order by timestamp desc

Run that query in front of your auditor once and most of the "but can we prove what it did" conversation ends.

11. A 30/60/90 pilot that survives contact with your CAB

Phase	Enabled capability
1	Detect Azure Monitor alerts
2	Create or correlate ServiceNow incidents
3	Perform read-only investigation
4	Add sanitized findings to ServiceNow
5	Recommend remediation without execution
6	Execute bounded actions after approval in Review mode
7	Validate technical and business recovery
8	Prepare incident resolution and follow-up records
9	Consider autonomy only for proven low-risk actions

Phases 1–5 have zero production write risk and deliver most of the MTTR reduction. Do not rush past them to get to the demo-friendly part.

The best first five candidates

App Service deployment-slot rollback — clean trigger, reversible action, unambiguous validation
AKS deployment rollback — same shape, one GitOps caveat
VMSS unhealthy-instance replacement — stateless, bounded, easy precondition check
Restricted VM disk-recovery runbook — high toil, high value, forces you to build the runbook pattern properly
Automatic ServiceNow incident creation and timeline updates — the compounding one; every incident from here on is better documented than any incident before it

These five share the properties you want: clear triggers, tightly bounded actions, measurable validation criteria, and practical escalation paths.

What to measure in week one

Before you enable a single write action, capture your baseline:

Median time from alert to first accurate human diagnosis
Percentage of incidents where the first hypothesis was wrong
Median time from diagnosis to mitigation
Percentage of incidents with a complete timeline in the ticket
Percentage of incidents that produced a follow-up problem record

The read-only phase moves the first, second, and fourth of those immediately. If it doesn't, your telemetry is the problem, not the agent — and that's a genuinely useful thing to discover in week one rather than week twelve.

12. Measuring whether it's actually working

Under Monitor → Incident metrics:

Metric	What it shows
Incidents reviewed	Total incidents the agent processes
Mitigated by agent	Resolved autonomously
Assisted by agent	Agent helped; a human completed it
Mitigated by user	Human resolved using agent-provided information
Pending user action	Waiting on a human

The counter-intuitive read: "Assisted by agent" and "Mitigated by user" are the healthy numbers during a pilot. A high "Mitigated by agent" count in month one means someone left autonomy on.

Watch Pending user action closely. A growing queue there means either your approval routing is broken or the agent is proposing things nobody is comfortable approving — both are important signals, and both are invisible without this dashboard.

Also check Monitor → Session insights periodically. Each insight card links back to the thread that generated it, so you can trace any learned pattern to its origin. If the agent has learned something wrong, this is where you find it — and #forget is how you fix it.

13. Resources

Core documentation

Incident response

Extensibility

Closing

The framing that makes this work isn't "AI runs my production." It's this:

Your agent is the most junior person on the rotation — and the most thorough. It will never skip the Resource Health check. It will never forget to compare against the previous slot. It will never write "restarted it, seems fine" in a work note at 4 AM. And it will never, ever be allowed to rm -rf anything.

Every guardrail in this post exists to keep it in that role. Scope the identity to resource groups. Keep production writes in Review. Wrap guest-OS work in runbooks with allowlists. Never let it purge a queue or replay a dead-letter message on its own. Keep the incident open until customers are actually served, not until the graphs look nice.

Do that, and the ten workflows above stop being a slide deck and start being your Tuesday.

Start with use case #1. One App Service, one slot, one alert rule, one response plan in Review mode. Watch it assemble an evidence chain you'd have spent twenty minutes building by hand, and then decide how much further you want to go.

The ten scenarios in this post are target response patterns. Each one requires appropriate telemetry, scoped Azure RBAC, response-plan instructions, approved remediation tooling, and ITSM integration. Confirm current Azure SRE Agent and ServiceNow connector capabilities against Microsoft documentation before implementing — the product is moving quickly, and several capabilities referenced here are in preview.

Stop paying for idle VMs — safely: ringed start/stop waves for your Azure estate

zmustafa — Wed, 26 Aug 2026 14:49:01 GMT

Turning non-production VMs off at night is the easiest Azure saving there is. Doing it without causing an outage is the part nobody writes about. This is an open-source scheduler that takes that saving as an application, in ordered rings, with two independent safety gates between you and a real power action — and it runs entirely in your own subscription.

It is 3am. Somewhere in your subscription, a dev estate is fully powered and doing absolutely nothing. Nobody has logged in since 18:40. The bill does not care.

The fix looks trivial for about ten minutes. Write a script, stop everything at 19:00, start everything at 07:00, collect the applause. Then someone runs it against an environment where the database tier and the app tier are just two more entries in the same resource group, the app tier comes up first, spends four minutes failing its health probe, and the platform team spends the morning explaining why the cost-saving initiative caused an incident.

The saving is easy. The sequencing is what breaks.

The whole application, end to end. Everything you are about to see runs against the built-in demo estate with both safety gates off — note the mock-mode banners.

Azure VM Scheduler is an open-source, MIT-licensed scheduler that models your estate the way you actually talk about it — applications hold rings, rings hold virtual machines — and fans every scheduled occurrence out into an ordered, staggered wave of per-VM actions. Starts walk the rings forward. Stops unwind them in reverse. And until you deliberately flip two separate switches, none of it touches a real machine.

Two problems, not one

The money

Deallocating a VM stops the compute meter. That is the whole trick, and it is a good one: a machine that only needs to run 07:00–19:00 on weekdays is genuinely needed for 60 of the 168 hours in a week — about 36%. The other 64% is compute you are buying and nobody is using.

Be precise about what you keep paying for, because this is where "we'll save 64%" quietly becomes a credibility problem in your next FinOps review: deallocation stops compute charges, not storage. Managed disks keep billing whether the VM is running or not, as do reserved public IP addresses and anything else with its own meter. The saving is real and it is large, but it is a saving on the compute line, not on the invoice total.

The shape

Azure already gives you several ways to turn a machine off on a timer, and they work. The catch is that they are shaped like a machine, or like a tag, and applications are not shaped like either. An application has an order. The data tier comes up before the app tier; the front door comes up last; and on the way down it all has to happen in reverse. A canary ring exists precisely so that it is the first thing to come up and the last thing to go down.

None of that is expressible as "shut this VM down at 19:00", because the unit is wrong. What a large estate needs is a scheduler whose unit of scheduling is the application, with ordered stages inside it — and that is a modelling problem, sitting on top of primitives Azure already exposes very well.

Where this fits alongside what Azure already gives you

Azure is not short of building blocks here. Azure Resource Graph will tell you what is deployed across every subscription you can see. Azure Resource Manager exposes start, deallocate and powerOff as first-class operations with proper RBAC around them. Azure Policy, Azure Automation and the platform's own auto-shutdown all sit on that same foundation. This project is not an alternative to any of that — it is a client of it. Discovery goes through Azure Resource Graph, power actions go through ARM, and identity goes through Microsoft Entra ID.

What it adds is a shape:

Tool	Designed for	Unit of action
VM / DevTest Labs Auto-shutdown	A single VM's daily shutdown time	One VM
Azure Automation runbooks	Arbitrary automation you author and maintain	Whatever you script
Start/Stop VMs v2	Scheduled, sequenced and CPU-triggered start-stop across whole scopes	Subscription, resource group or VM list
Azure VM Scheduler	Ordered, application-aware start/stop waves with per-action safety gates	Application → ring → VM

Read that table honestly. If a resource group is a good enough boundary for your estate, use the built-in option — it is less to run, less to patch and less to explain to your auditor. Plenty of dev estates genuinely are flat, and for those this project is over-engineering.

This exists for the estates where it isn't. Where boot order matters, where a canary ring means something, where "stop everything in rg-dev" would take down the one machine that runs overnight settlement, and where the blast radius of a scheduling mistake is measured in incidents rather than in dollars.

A closer look at Start/Stop VMs v2

Of the built-in options, Start/Stop VMs v2 is the nearest neighbour, so it deserves a proper description rather than a table row. It is a Microsoft-published solution you deploy into your own subscription: an Azure Functions app holding a managed identity in Microsoft Entra ID, five Azure Logic Apps that carry the schedules and call that app with a JSON payload, Azure Storage for execution metadata and queues, and Application Insights behind a shared Azure dashboard with optional email through an action group. Each action is scoped to one or more subscriptions, resource groups, or an explicit VM list — with wildcard exclusions — and machines are ordered within a scope by tagging them sequencestart and sequencestop with values from 1 to N. It also does something this project does not: AutoStop watches CPU utilisation and stops idle machines via an alert.

One thing to know if you are choosing today: Microsoft's guidance states that no further development or enhancements are planned for Start/Stop VMs v2 beyond what is needed to keep its components on supported versions. That is ordinary platform evolution, not a warning — it still deploys, it was moved to the .NET 8 isolated worker model in 2024, and it still does what it does well.

	Start/Stop VMs v2	Azure VM Scheduler
Where the schedule lives	Azure Logic Apps you deploy and manage	The application's own database, edited in a UI or over its API
How machines are grouped	Subscriptions, resource groups or a VM list, with wildcard exclusions	Application → ring → VM, inherited down the tree
How order is expressed	A sequencestart / sequencestop tag per machine, processed in ascending order	The ring's sequence, inherited by every machine in it
Where results land	Application Insights, a shared Azure dashboard and action-group email	Per-run and per-attempt records in the app, with per-attempt retry
Where it runs	Azure	Azure, or a laptop on SQLite

Three differences follow from that, and they are differences of design centre rather than of quality.

Grouping is modelled, not enumerated. A scope in Start/Stop VMs v2 is written into the payload of the Logic App that owns it. Here the scope is a thing in the model: schedule the application and every ring and machine underneath inherits it, so adding a VM to a ring puts it in the right wave with nothing else to edit and no tag to remember.

Order and overlap are properties of that model. Sequencing tags are processed in ascending order for both directions, so a reverse-order shutdown is something you author machine by machine in the sequencestop values. Here reverse is the default for stops — the wave diagram further down this post is two rows of the same list read in opposite directions. And because schedules can be inherited, they can overlap, which is why resolution is per action and nearest-wins: a property you need only once inheritance exists.

The guards are sized to the unit. When the thing you point a schedule at is an entire application, a mistake is proportionally bigger — so the stop path gets two independent gates, never_stop that a machine inherits from any ancestor rather than an exclusion list per schedule, and an exact-count confirmation. That machinery is the price of the larger scoping unit, not a criticism of a smaller one.

So: if you want CPU-triggered auto-stop, availability in the US Government cloud, a Microsoft support path, or simply nothing extra to run beyond Azure resources themselves, Start/Stop VMs v2 is the better answer. This project is for the estates where ordering, inheritance and blast radius are the hard part.

Start/Stop VMs v2 details verified against its overview documentation, July 2026.

Model the estate the way you talk about it

The hierarchy is deliberately, aggressively shallow: an application holds rings, a ring holds virtual machines, and that is the entire tree. Exactly two levels, enforced everywhere — on create, on move, on CSV import, on settings import. There is no way to end up with a ring inside a ring inside a ring, which means there is never a debate about what a given schedule actually covers.

The built-in demo estate shows the idea in about thirty seconds. Zava Commerce has three rings — Canary (one VM), Pilot (two), Production (four). Zava Analytics has Batch and Interactive. Zava Intranet has Pilot and Production. If your estate is tiered rather than ringed, name them Data, App and Web instead; the model does not care what a stage is called, only that stages are ordered.

Attach a schedule to the application and every ring inherits it. Override one ring and only that ring changes. Override a single VM and only that machine changes. That is the whole inheritance story, and it is short on purpose.

The ring board for one application. The sequence number on each ring is the whole ordering model — there is nothing else to configure.

Waves: what actually happens at 06:30

One occurrence produces one run. One run fans out into one attempt per virtual machine, ordered by ring sequence, with a configurable stagger between machines so you never hand ARM several hundred simultaneous power requests and watch it start throttling you.

Zava Commerce's start wave fires at 06:30 with a 60-second stagger. The canary machine goes first. A minute later the two pilot machines. A minute after that the four production machines, one per minute. Seven machines, seven attempts, in a defined order, each recorded.

The stop wave at 20:00 does the same thing backwards. Stops default to reverse — the last ring in the sequence is the first one down, and the canary ring, which came up first, goes down last. That default is not a convenience; it is the single most important behaviour in the product, because reverse-order shutdown is exactly what a hand-rolled script forgets.

One application, two waves. Read the stop row right to left and you have the start row — that is the point. If you only take one picture from this post, take this one.

Nearest schedule wins — per action

Here is the guarantee that makes overlapping schedules safe to live with. For each action independently, a VM is acted on by the nearest schedule that targets it: a schedule on the VM beats one on its ring, which beats one on its application. Deeper shadows shallower.

Because start and stop are resolved separately, a machine ends up with at most one effective start and at most one effective stop. Stack an application-wide 06:30 start, a ring-level 07:15 start and a per-VM 05:00 start on the same machine and it still starts exactly once, at 05:00. It cannot be started twice. It cannot be stopped twice. That property holds no matter how many schedules sit above it, which is what lets you layer overrides without keeping a mental model of the whole tree.

A wave running. Each row appears as its machine's turn comes up, in ring order, one stagger interval apart.

Every wave leaves a paper trail

Every wave is fully reconstructable afterwards. A run carries a rolled-up status — succeeded, partially_failed, failed, timed_out or cancelled — and each attempt records its own status, whether it ran in real or mock mode, a message, its attempt number, its sequence position and a correlation id. When someone asks "what happened at 06:30 on Tuesday", the answer is a page, not an investigation. Failed attempts can be retried individually, or the whole run can be retried and it will pick up only the ones that failed.

One wave, after the fact: the mode it ran in, the correlation id, the roll-up, and every machine it touched. This page is the answer to "what happened at 06:30 on Tuesday".

Recurrence you can check before you trust it

Schedules come in four flavours: one-time, daily, weekly and full five-field cron — lists, ranges, steps, month and day names, Sunday as 0. Daily and weekly are stored as friendly fields and translated to cron behind the scenes, so there is exactly one occurrence engine in the codebase rather than one engine and three special cases that disagree at the edges.

Everything is timezone-aware and stored against an IANA zone (the demo estate runs on America/New_York). Times are wall-clock: 08:00 stays 08:00 across a daylight-saving change, which is what an operations team means when they say "eight in the morning". And an occurrence that lands in a spring-forward gap — an 02:30 job on the morning the clocks jump from 02:00 to 03:00 — is skipped, not silently shifted to 03:30. That is a decision, made once, applied consistently, and covered by tests.

The preview is the point

Cron is where scheduling bugs live. Everybody has confidently written 0 0 * * 0 and then argued about which day that is.

So the editor does not ask you to trust it. On every keystroke it asks the server to describe the recurrence in plain English and return its next five occurrences — and the server answers using the same engine that will actually fire the schedule. Not a browser re-implementation of cron that agrees with the backend most of the time. The same code path. What the editor shows you is, by construction, what the scheduler will do.

Watch the right-hand panel as the cron expression is typed. That description and those five dates come back from the server on every keystroke — from the same engine that fires the schedule.

Schedules that know when to stop

A schedule can carry start_date and end_date as local calendar bounds, and a run_limit budget. When the budget is spent or the end date passes, the schedule flips to status completed and stops producing a next run — rather than sitting there looking enabled while silently never firing again, which is the failure mode that has you debugging a scheduler that is working perfectly. Manual runs do not consume the budget; only scheduler-triggered ones do.

Why you can point this at production without flinching

This is the section the rest of the post exists for. Seven guarantees, each one a mechanism rather than a promise.

The gates

Two independent gates.A real Azure start requires the globalENABLE_REAL_AZURE_STARTS setting and the target tenant's allow_vm_start permission. A real stop requires ENABLE_REAL_AZURE_STOPS and allow_vm_stop. They are entirely separate. Turning on starts does not, and cannot, enable stops.
Re-evaluated on every single attempt.The permission decision is deliberately kept apart from the expensive work of building an ARM credential. Adapters get cached because acquiring a token is costly; thedecision is remade for every attempt. Revoke a tenant's stop permission while a stop wave is halfway through and the remaining attempts stop being real — immediately, not whenever a cached adapter happens to expire.
Mock-first.Until both switches are on, every wave runs against a deterministic mock adapter that records a simulated result. You can build an entire estate, wire up every schedule, preview the next month of occurrences, run the waves, inspect the attempts and show the whole thing to your change board without touching a single machine. A fresh deploymentarrives inert — both gates ship false.

The gates are a visible posture, not a buried environment variable. This is what a fresh deployment looks like, and it is what it keeps looking like until somebody changes it.

The guards

never_stop.Set it on a VM, or on any ancestor, and that machine is removed from every stop wave and every on-demand stop. Zava Payments demonstrates it: its Production ring runs overnight settlement, so it is marked never_stop and the application has a start wave but no stop wave at all. Nothing you do to the parent schedule can pull those machines into a shutdown.
Exact-count confirmation.Selecting machines and pressingStop now stops nothing. The dialog names the tenant, states whether it will deallocate or power off, and refuses to arm until you type the machine count back. It is unglamorous and it has saved somebody's afternoon.

Eighteen machines selected, but the dialog offers to stop sixteen — two are never_stop and it says so. The confirm button stays dead until 16 is typed. Three of the seven guarantees, in one dialog.

Conflict guard.The scheduler skips an attempt whose VM already has an opposite-action attempt in flight. If a start wave is still working on a machine, a stop wave will not race it.
Read-only tenants can never power anything.A connection marked read-only refuses start and stop outright, gate or no gate, and a disabled connection refuses everything.stop_mode is deallocate by default — the one that actually stops the compute meter — with power_off available per schedule when you need the machine halted but not released.

Coverage gaps: the report you didn't know you needed

Any scheduling system's second-order failure is partial coverage. Not "the scheduler broke" — the scheduler is fine — but "eleven machines quietly fell outside it". So the overview goes looking for exactly that, in both directions:

Starts but never stops — machines with an effective start wave and no effective stop wave. These come up every morning and never go down. They are the ones burning money while everyone congratulates themselves on the savings.
Stops but never starts — the inverse, and the more expensive mistake. These go down tonight and stay down, and you find out at 09:00 tomorrow.
Stop protected — machines excluded by never_stop, listed explicitly so the exclusion is a visible decision rather than something you rediscover during an incident.

It also flags start and stop waves that overlap in time, including the stagger tail, so you find out at design time that your 20:00 stop wave for a forty-machine application is still running when the 20:30 start wave for something else begins.

Gaps in both directions, named machine by machine. The middle one — three machines that stop tonight and are never started again — is the one that ruins a Monday.

The same page carries the pre-flight readiness checks (both global gates, tenants that are read-only or missing the permission a live schedule needs, credentials approaching expiry), windowed KPIs with a previous-period delta, a fourteen-bucket trend, the next-24-hours strip, the rollout plan, an application health matrix and reliability statistics.

Getting a real estate in without a spreadsheet

Nobody types four hundred resource IDs. Three routes in.

Paste bare VM names

Give it vm-commerce-prod-01 and it resolves the subscription and resource group for you through Azure Resource Graph. If a name is ambiguous across subscriptions, it surfaces the candidates and asks you to pick rather than guessing. Duplicates are blocked. You can also browse a subscription and select machines directly, or paste full resource IDs if you already have them.

Four names in. Three are already filed and it tells you exactly which ring holds each; the fourth is unknown and gets offered a tenant to resolve against.

Import VMs

Point it at a UTF-8 inventory CSV and it runs a validating preview that reports exactly what would be created, updated and skipped before anything is written. The preview is bound to an encrypted, expiring token, and the commit is atomic — a stale or tampered preview is rejected rather than half-applied. The simplest valid file is a single column of VM names.

Nothing is written during a preview. The commit is all-or-nothing, and a preview that has gone stale is refused rather than partly applied.

Who gets to press the button

Access control is capability-based, and it is enforced on the API rather than hidden in the UI.

Users hold roles directly or through access groups; their effective permissions are the union of both, resolved once per request. Five roles ship built in — admin, operator, auditor, viewer, noaccess — and they are re-seeded on every start, so a capability added in a new version is never left unusable on an existing deployment. Custom roles are free-form on top.

Sign-in is local, or SSO, or both. SSO is multi-provider: any number of OIDC issuers configured from their discovery documents (Microsoft Entra ID is just the case where the issuer is derived from a directory id), and SAML 2.0 verified with signxml — where the code reads only the signed subtree the library returns, never the raw document, which is how a whole family of SAML signature-wrapping attacks stops being interesting.

Three walls are enforced for every request: a noaccess allowlist, a forced-password-change allowlist, and a database-backed per-IP brute-force throttle that catches one attacker spraying many usernames rather than only many attempts at one account. Two lock-out guards return 409 rather than letting you strand yourself: you cannot remove the last enabled account that can manage users, and you cannot disable local sign-in when no SSO provider is enabled.

Every action lands in an audit log.

Users, roles, access groups, live sessions, sign-in policy and the SSO providers — one page, one permission to reach it.

Where it runs

One container image. The built single-page app is served by FastAPI at the same origin, so there is no CORS story, no second container and no reverse-proxy configuration to get wrong. The static mount is registered last, so it can never shadow the API.

The Deploy to Azure button provisions, in your subscription: a Container App running the public image, Azure Database for PostgreSQL flexible server (Burstable B1ms), an Azure Files share mounted at /app/.data for the encryption key and connection registry, a Container Apps environment and a Log Analytics workspace. Replicas are pinned to one, because the scheduler is in-process by design. Setting privateNetworking at create time injects the environment into a VNet and puts both the database and the storage account behind Private Endpoints with public access disabled.

Everything one deployment creates. The private-networking switch is create-time only — an existing public deployment has to be redeployed, not flipped.

Connecting Azure without storing a secret

The Container App gets a system-assigned managed identity. Grant it Reader on the scope you want to manage plus start / deallocate / powerOff on the target VMs, add a tenant in the app using the default_chain auth method, and no credential is stored anywhere at all. Where you do store client secrets, they are Fernet-encrypted at rest and never leave the host.

Two tenants, one writable and one read-only, both authenticating as the host identity. The per-tenant permissions are the second half of every gate.

Or run the whole thing on a laptop against SQLite. DATABASE_URL picks the engine; nothing else about the application changes.

How it's judged

The claims above are mechanisms, and each one is testable:

A VM is never started twice nor stopped twice, whatever the schedule overlap — resolution is per action, and the nearest schedule wins.
The recurrence preview is produced by the same engine that fires the schedule, so what the editor shows is what the scheduler does.
An occurrence in a DST spring-forward gap is skipped, never silently shifted.
A real Azure power action requires two independent switches, re-checked on every attempt — so revoking a permission halts a wave already in progress.
never_stop is honoured by scheduled stops and on-demand stops alike.
Every wave is reconstructable after the fact: run status, per-VM attempt, mode, message, attempt number, sequence position and correlation id.
A fresh deployment cannot touch Azure until a human deliberately arms it.

What it deliberately isn't

Worth knowing before you invest an afternoon:

Single-replica by design. The scheduler runs in-process and the Bicep pins replicas to 1/1. That is a conscious trade — no leader election, no distributed lock, no split-brain — but it does mean this is not a highly-available control plane.
Virtual machines only. Not scale sets, not AKS node pools, not App Service plans, not SQL.
Not a cost tool. It will not price your savings, forecast them or chargeback them. It turns machines off; your existing cost tooling reports on the result.
Two levels, permanently. If you need a five-deep hierarchy, this is the wrong shape and no amount of configuration will change that.

Try it

The fastest honest test is to run it against nothing. Deploy it with the button below — it arrives with both action gates off, so it physically cannot touch a machine — then open Settings → Demo data and load the sample Zava estate: four applications, nine rings, eighteen virtual machines and seven start/stop waves, complete with a never_stop ring and an application that starts but never stops, so the coverage-gap detection has something to find. It loads in one click and removes exactly what it created.

Build your real rollout on top of that, rehearse every wave against the mock adapter, and only then decide whether to arm anything.

Let GitHub Copilot do the setup

Open an empty folder, put Copilot Chat in agent mode, and hand it the repository:

Clone https://github.com/zmustafa/AzureVMScheduler into this folder, set it up for local development, then start the backend and the frontend.

That is enough. The repository ships a README with the setup steps and VS Code tasks for both servers, so the agent has something to follow rather than a command line to invent. The API comes up on 127.0.0.1:8000 and Vite on 127.0.0.1:5173 proxying /api to it; with no DATABASE_URL set, the whole thing runs on a SQLite file under .data/ with nothing else to install. Sign in as admin, change the password when it makes you, and load Settings → Demo data.

Repository: https://github.com/zmustafa/AzureVMScheduler
Container image: https://hub.docker.com/r/zmustafa/azure-vm-scheduler
Licence: MIT — issues and pull requests welcome

If you try it, I would genuinely like to hear which part of the safety model you found excessive and which part you found insufficient. That is the argument worth having.

Original post.

This is a community open-source project and is not affiliated with or endorsed by Microsoft.

Navigating the Azure Databricks CSP Mandate

anishekkamal — Wed, 26 Aug 2026 14:47:42 GMT

Mandatory Deadline: The Compliance Security Profile (CSP) becomes mandatory for processing HIPAA, HITRUST, and IRAP regulated data on Azure Databricks by September 1, 2026.
Permanent Change: Enabling CSP is an irreversible, workspace-level configuration. Reversion requires deleting and recreating the workspace, emphasizing the need for meticulous pre-enablement testing.
Architectural Impact: CSP introduces significant changes, notably requiring VNet encryption which can disrupt existing on-premises connections, and disabling partner-powered AI features by default.

The evolving regulatory landscape demands rigorous security and compliance measures for data processing in the cloud. For organizations leveraging Azure Databricks, the introduction and upcoming mandatory enforcement of the Compliance Security Profile (CSP) represent a pivotal shift. This profile is not merely a feature toggle; it's a foundational architectural commitment designed to harden your Databricks environment against stringent compliance standards. Understanding its implications and preparing proactively is crucial to ensure uninterrupted operations and continued regulatory adherence.

Why CSP Matters

The Compliance Security Profile (CSP) is a robust, platform-hardening configuration applied at the workspace level within Azure Databricks. Its primary objective is to facilitate adherence to a broad spectrum of compliance standards by enforcing additional monitoring, utilizing hardened compute images, ensuring inter-node encryption with specific instance types, and implementing stricter controls across Databricks workspaces. This provides a secure baseline for the data plane, incorporating enhanced security monitoring capabilities.

For organizations handling sensitive data, such as Protected Health Information (PHI) under HIPAA, or data subject to HITRUST and IRAP regulations, CSP is not just beneficial—it is becoming mandatory. Starting September 1, 2026, all Azure Databricks workspaces processing data under HIPAA, HITRUST, or IRAP will be required to have CSP enabled. Failure to comply will impede the ability to process such regulated workloads.

Conceptual diagram illustrating data protection and complianceonDatabricks.

Key Compliance Standards Supported by CSP

The compliance security profile supports or is required for processing data under a wide array of global and industry-specific compliance standards. This comprehensive coverage underscores its importance for organizations operating in highly regulated sectors:

C5
CCCS Medium (Protected B)
FedRAMP
HIPAA (Health Insurance Portability and Accountability Act)
HITRUST (Health Information Trust Alliance)
IRAP (Infosec Registered Assessors Program)
ISMAP
Korean Financial Security Institute (K-FSI)
PCI-DSS (Payment Card Industry Data Security Standard)
TISAX
UK Cyber Essentials Plus

Understanding CSP's Architectural Implications

One of the most critical aspects of CSP to grasp is its permanence. Once enabled on a workspace, or once any regulated data has been processed within a CSP-enabled workspace, the profile cannot be simply toggled off. To revert to a non-CSP state, you would need to delete the existing workspace and provision a new one without the CSP activated. This irreversible nature necessitates a thorough planning and validation phase before implementing CSP in production environments.

Cost Considerations: The Enhanced Security & Compliance Add-on

Enabling CSP automatically activates the "Enhanced Security and Compliance add-on" billing, which will incur additional costs. Organizations must factor these increased expenditures into their budgeting and financial planning. Reviewing the Azure Databricks pricing page is essential to understand the full financial impact.

Networking Challenges: VNet Encryption and On-Premises Connectivity

A significant "gotcha" highlighted in various sources is the impact of CSP on network architecture. CSP enforces stricter network security policies, often requiring VNet encryption. If your Azure Databricks workspaces utilize VNet Injection and connect to on-premises systems via ExpressRoute or VPN, this VNet encryption can potentially disrupt existing connections. This necessitates close collaboration with networking teams to verify compatibility, test connectivity in pre-production environments, and potentially reconfigure network components such as routing tables, security groups, or firewall rules.

Feature Impact: AI Assist and Public Preview Limitations

To maintain stringent compliance boundaries, CSP workspaces alter default behaviors and restrict certain features:

Partner-powered AI Features: By default, "Partner-powered AI features," which include some AI-powered coding assistants like Genie Code, are disabled in CSP workspaces. While Databricks-hosted models may power Genie Code in CSP workspaces for specific compliance standards (e.g., HIPAA, FedRAMP Moderate, PCI-DSS, HITRUST) in certain regions, organizations relying on these features should assess alternatives or plan for explicit enablement, understanding the potential constraints.
Preview Features: Only a limited set of Public Preview features are supported in CSP workspaces. Workloads dependent on unsupported preview features may face limitations or require adjustments.

Preparing for the CSP Mandate

To ensure a smooth transition to CSP, a detailed and phased approach is recommended. This involves thorough assessment, strategic planning, rigorous testing, and clear communication across various teams.

The radar chart above visualizes the current readiness level (Pre-CSP Readiness) versus the desired target state (Post-CSP Target State) across critical preparation areas for the Compliance Security Profile. It underscores the significant effort required in areas like network reconfiguration and thorough testing to achieve full compliance and operational stability.

Inventory and Assessment
- Identify Regulated Workspaces: Catalogue all existing Azure Databricks workspaces. Determine which ones currently process, or are planned to process, data subject to HIPAA, HITRUST, or IRAP.
- Review Data Pipelines: Map out all data ingress and egress points for these identified workspaces, including connections to on-premises data sources, other cloud services, and external APIs. This helps identify potential network impacts.

Network Architecture Review

- Evaluate VNet Encryption: If your workspace uses VNet Injection and has private connectivity to an on-premises network (e.g., via ExpressRoute or VPN), verify that these connections will remain functional under CSP's new encryption requirements.
- Plan for Reconfiguration: Allocate resources for potential network re-architecting, including updating routing tables, security groups, or firewall rules to accommodate encrypted traffic flows.

Feature Impact Analysis

- AI & Assist Features: Assess the reliance of your development teams on "Partner-powered AI features" like Genie Code. Determine if Databricks-hosted alternatives meet their needs or if explicit enablement (and its associated compliance considerations) is required.
- Preview Features: Consult official documentation to check for limitations on Public Preview features in CSP workspaces and plan accordingly if your workloads depend on any unsupported features.

Data Handling Responsibilities

- Sensitive Information Fields: Implement policies to prevent sensitive data from being entered into customer-defined input fields (e.g., workspace names, compute resource names, tags, job names, URLs), as these might be stored or processed outside the compliance boundary.
- Encryption at Rest: Verify that all data containing PHI or other regulated information is encrypted at rest in any storage location interacting with Databricks, including workspace storage accounts.

Execution and Validation Plan

- Staging Environment First: Always enable CSP on a clone of your production environment in a staging subscription before rolling out to production.
- Thorough Testing: Conduct comprehensive end-to-end tests on the staging workspace to validate data pipelines, network connectivity, user workflows, and performance.
- Enablement Method: Choose the appropriate tooling for enablement—Azure Portal, Azure CLI, PowerShell, ARM templates, or Terraform—to ensure consistency and automation.

Detailed Enablement Methods

Enabling CSP can be done for both new and existing workspaces using various Azure management tools. Regardless of the method chosen, it's crucial to select the applicable compliance standards during enablement.

Enabling CSP on New Workspaces

For new workspaces, the compliance security profile settings at the account level can control whether it's enabled by default. Account administrators can enable CSP individually during deployment using:

Azure Portal
Azure CLI (e.g., az databricks workspace create --enable-compliance-security-profile --compliance-standards '["HIPAA"]')
PowerShell
ARM templates
Terraform (using the databricks_compliance_security_profile_workspace_setting resource)

Enabling CSP on Existing Workspaces

Existing workspaces can also have CSP enabled through similar methods:

Azure Portal: Navigate to Workspace > Settings > Security & compliance > Enable compliance security profile; select standards and save.
Azure CLI / PowerShell / ARM templates / Terraform: Utilize these tools to modify the workspace settings and enable CSP, specifying the relevant compliance standards.

Understanding Key Technical Considerations

Beyond the general planning, specific technical details must be understood and addressed to ensure a successful CSP implementation.

Account-Level Requirements

Before enabling CSP, ensure your Databricks account meets these prerequisites:

Your Databricks account must include the Enhanced Security and Compliance add-on.
Your Databricks workspace needs to be on the Premium pricing tier.

Security Configuration Table

The table below summarizes critical security configuration aspects influenced by CSP:

Security Aspect	Pre-CSP State	Post-CSP State	Preparation Action
Compute Images	Standard images	Hardened compute images	No direct action required, platform handles.
Inter-node Encryption	Configurable/Optional	Enforced (for specific instance types)	Ensure use of supported instance types.
Network Encryption (VNet)	Standard VNet configuration	VNet encryption required	Test on-premises connectivity, reconfigure network routes/firewalls if needed.
AI Assist Features	"Partner-powered AI features" typically enabled	"Partner-powered AI features" disabled by default	Assess reliance, plan explicit enablement or alternative Databricks-hosted models.
Monitoring	Standard monitoring	Enhanced security monitoring, additional agents	Familiarize with new monitoring capabilities and data.
Compliance Scope	Self-managed controls	Platform-enforced controls for selected standards	Select appropriate compliance standards during CSP enablement.

To further understand the broader context of security and compliance on Databricks, including how CSP fits into the larger ecosystem, watch this video from Databricks: Databricks Security & Compliance

References

Compliance security profile: https://learn.microsoft.com/en-us/azure/databricks/security/privacy/security-profile
Configure enhanced security and compliance settings: https://learn.microsoft.com/en-us/azure/databricks/security/privacy/enhanced-security-compliance
HIPAA, Azure Databricks, Microsoft Learn: https://learn.microsoft.com/en-us/azure/databricks/security/privacy/hipaa
What is Azure Virtual Network encryption: https://learn.microsoft.com/en-us/azure/virtual-network/virtual-network-encryption-overview

Building an AI Azure Ops Workbench — Part 3: Reverse engineer Azure Architecture Diagrams

zmustafa — Wed, 26 Aug 2026 14:50:20 GMT

In Part 1 we made the case for an operations copilot that runs inside your own tenant. In Part 2, the Azure Support Agent's war room of specialist agents root-caused a live 5xx in about ninety seconds. But every one of those specialists had to be pointed at something first — and defining that "something" is the problem this post solves. Welcome to the second pillar: Map.

Here's a question that sounds trivial and almost never is: which resources make up your application?

Azure knows. It exposes the answer generously, too — Azure Resource Graph will return every resource in the tenant, with full properties, from a single KQL query. The Portal shows every relationship if you click far enough down. Azure Monitor holds the metrics, Microsoft Entra ID holds the identities. Very little about your estate is genuinely hidden.

What's scarce isn't data. It's time. Nobody has a spare afternoon each week to re-derive which forty of your eleven hundred resources constitute "the checkout app," redraw the diagram, and write down why the private endpoint is there. So the map drifts out of date — not because the information went missing, but because keeping it current is work that never wins against a sprint. That's a bandwidth problem, and bandwidth problems are exactly the kind of thing you can hand to an agent that never gets bored.

So the Azure Support Agent builds the map for you and keeps it alive. Five pieces, each feeding the next: Workloads that mean something, a live architecture diagram reverse-engineered from what's actually deployed, an Architecture Memory that remembers design intent, a Know-Me doc your on-call can read at 3 a.m., and an Estate Graph that renders the whole tenant as one navigable picture.

One promise up front, because it shapes every design decision below: mapping your estate never changes it. Every feature in this post is read-only against Azure.

Part one — Workloads

A unit of scope that actually means something

A subscription is an accounting boundary. A resource group is a lifecycle boundary. Neither one reliably equals "an application." So the workbench introduces one that does: the Workload.

A workload is a named set of scopes, and it can mix all four levels Azure gives you — a management group, a subscription, a resource group, or an individual resource ID. Real estates are untidy, so membership is subtractive as well as additive: every scope carries an excludes list, which lets you express "this entire resource group except that one legacy VM" without hand-listing thirty-nine resource IDs and re-listing them next month.

Alongside its scopes, a workload carries the metadata that everything downstream reasons about: workload type, environment, business criticality, data classification and free-form tags. That metadata isn't decoration — the fleet views pivot on it, and the assessments in Part 4 weight their findings by it.

This layer matters more than it first appears, because the workload is the unit of scope for the entire product. Every assessment, every coverage report, every architecture diagram in this post, and every deep investigation from Part 2 is scoped to one. Get this right and everything downstream inherits it. Get it wrong and you're running beautiful analysis on the wrong set of resources.

Autopilot: let the AI propose the boundaries

Defining workloads by hand is perfectly fine for five of them. It does not survive eleven hundred resources, and it really doesn't survive an estate somebody else built.

Autopilot reads the estate and proposes the groupings, reasoning over exactly the signals an experienced engineer would use: tag values, naming conventions, resource-group boundaries, and which resources actually sit together. It's built for real tenants rather than demos — a discovery pass handles up to 5,000 resources (and tells you when it truncated rather than silently sampling), feeding the model in batches of 500 in a map-reduce pattern so a large estate never has to fit inside one context window.

The important part is what happens next: nothing is saved behind your back. Autopilot streams out candidate workloads, each with its reasoning, a confidence level, and a breakdown of what's inside it — and then it stops. Those candidates are proposals until you explicitly save them. You can rename, merge, split, or bin them. This is the same posture as the Launch gate in Part 2: the AI proposes, a human disposes.

Each Autopilot-born workload also remembers its origin — the management group or subscription it was discovered from — which powers a one-click Refresh when the estate moves on, instead of making you re-run discovery from scratch and reconcile by hand.

Groups, for when one app is really five

Most real applications exist several times over: CRM PROD, CRM DEV, CRM DR. Treating those as three unrelated workloads is technically correct and practically useless.

Groups are a non-destructive association on top of workloads — members keep their own identity, their own scopes and their own health score, but you can see them as one family. And because the naming pattern is so consistent in practice, the workbench will suggest the groupings for you: the suggestion pass strips environment tokens (prod, dev, stg, test, dr, shared, sandbox) from workload names and groups whatever stems match. CRM PROD and CRM DEV both stem to CRM, so they're offered as a group.

It's a small feature that does something disproportionately useful: it lets you ask "is this application healthy?" rather than "is this one environment of it healthy?"

A health score you're allowed to argue with

Each workload the Azure Support Agent tracks carries a composite health score out of 100. The interesting part isn't the number — it's that you can take the number apart.

The score is a weighted average of seven signals, and they are deliberately not weighted equally:

Signal	Weight	Why
Monitoring	1.0	Are the right alerts in place?
Telemetry	1.0	Are logs and metrics actually being collected?
Backup / DR	1.5	An unrecoverable workload is the worst failure mode on the list
Performance	1.0	Is anything saturating or trending at its limits?
Ownership	1.0	Does a human own this?
Policy	1.0	Is it governed and compliant?
Tags	0.5	Untagged is untidy, not unsafe

Bands are plain: 80 and above is good, 50 and above is a warning, below 50 is poor, and a workload with no analyzed signals at all reads unknown rather than zero.

Three design decisions are worth pulling out, because they're what make the number usable rather than decorative.

First, it explains itself. The API doesn't return a bare score; it returns the contributing signals, the missing ones, and the weights actually applied. "Why is this 62?" has a real answer you can put on a slide, not a shrug.

Second, unmeasured never masquerades as bad. Signals you haven't analyzed yet are excluded from the denominator entirely — a workload is never punished for a backup assessment nobody has run yet. It's marked unknown and the score is computed from what's actually known. A score that quietly conflates "bad" with "not yet measured" is a score people learn, correctly, to ignore.

Third, you can disagree. The weights are an admin setting. If your organisation's risk appetite says tags matter more than that, change it — the score is a tool for your judgement, not a replacement for it.

The board, the table and the cockpit

Fleet views tend to fail in one of two directions: a wall of rows nobody reads, or a dashboard so abstract it says nothing. The workbench offers three deliberately different lenses.

The table is the working view — one row per workload with environment pills, the score badge, the individual health signals broken out, resource counts and a risk column that folds in retirements and critical findings. It's what you scan on a Monday.

The board is a Kanban-style swimlane view grouped by environment — production, staging, development, test, DR, shared, unknown — which is the fastest way to spot the thing everyone dreads: something sitting in production with a health score in the reds.

The cockpit is the strip across the top: health distribution across the fleet, a composition treemap, an environment × criticality matrix, and a risk ticker. Four small pictures, each answering "where should I be looking?"

Underneath the visuals sits a shared resource taxonomy — every Azure resource type is classified into a category (compute, web, containers, data, storage, networking, security, integration, AI/ML, analytics, monitoring, management) and an architectural layer (edge, presentation, application, integration, data, networking, security, monitoring, shared). That single classification is what lets a donut chart, a treemap and an architecture diagram all speak the same language about the same estate.

And the charts themselves are honest little things: a composition donut by category, a seven-axis health radar (one axis per signal, so a lopsided workload looks lopsided), a sparkline of score trend, and a treemap of resource categories by weight. Click into a single workload and you get an Overview tab — at-a-glance, health and coverage, watchers, security, lifecycle — plus a full resource table, with chips that jump you straight to that workload's architecture, its retirement radar, its policy view.

Part two — The Azure Support Agent's Architecture Designer

Now draw it, from what's actually deployed

This is the part that tends to make people sit up in a demo, so it's worth going through properly.

Point the workbench at a workload, press Generate, and it reverse-engineers an interactive architecture diagram from live Azure data. What follows isn't one big AI call — it's a pipeline with a deliberate division of labour: Azure supplies the facts, the model supplies the reading of them, and code enforces the truth. That split is the whole reason the output is worth trusting.

Step one: read the estate, twice

The workbench resolves the workload's scopes into a set of subscriptions, resource groups and resource IDs, then queries Azure Resource Graph in two passes.

The first pass is deliberately cheap — a lightweight inventory projecting only what it needs to identify things: id, name, type, kind, location, resourceGroup, subscriptionId, sku, identity, zones and tags, plus the resource groups themselves.

The second pass goes back for the interesting part: properties, fetched in chunks of 30 resources at a time. Properties are where the actual topology hides — but they're also enormous and mostly irrelevant, so they're filtered before they ever reach the model. The filter keeps keys that carry relationships and drops the rest:

networkProfile · ipConfigurations · subnet · privateLinkServiceConnections · serverFarmId · storageProfile · agentPoolProfiles · vnetSubnetID · backendPools · routingRules · keyVaultProperties · connectionStrings · siteConfig · hostNames · primaryEndpoints

Two budgets bound the whole thing: roughly 6,000 characters per resource against a 120,000-character total. When the budget runs out, the remaining resources are explicitly marked as omitted rather than silently dropped — the pipeline would rather tell you it ran out of room than quietly draw you an incomplete picture. A sprawling workload degrades gracefully instead of blowing the context window, which is the difference between a feature that works on your estate and one that only works in a screenshot.

Step two: what the model is actually asked to do

The model receives the workload name, the resource count, and a compact JSON array of resources with their trimmed properties. The instruction that matters most is this one: infer relationships from the properties, not from what you know about Azure in general. A model asked "how do App Services usually connect to SQL?" will confabulate a plausible architecture. A model asked "here is serverFarmId and here is privateLinkServiceConnections[], what do they tell you?" is doing evidence-based reading.

It's asked to return a strict JSON contract — a name, a description, nodes[], edges[], groups[], a short rationale, and a confidence value. Nodes carry an arm_id tying them to a real resource. It's permitted at most a couple of conceptual nodes without an ARM ID — "Users", "Internet" — because a diagram with no origin for the traffic is harder to read, and that's a bounded, honest exception rather than a licence to invent.

The call is given generous headroom (16,000 tokens) so a large estate doesn't produce a diagram truncated mid-JSON, and if the model returns nothing it retries once before reporting failure rather than showing you an empty canvas.

Step three: the validation pass, where trust is actually won

This is the least glamorous stage and the most important one. Everything the model returns is treated as a proposal and put through normalization before it's allowed near your screen:

Groups are de-duplicated, and their kind is clamped to the permitted set — subscription, resource group, VNet, tier, custom.
Nodes are de-duplicated, and each arm_id is cross-referenced against the real inventory. Category and layer are re-derived from the actual ARM type rather than trusted from the model. Node metadata is capped at six key–value pairs with length limits, so nobody can stuff an essay into a diagram label. Group references that point at a group that doesn't exist are cleared.
Edges get the strictest treatment: any edge pointing at a node that doesn't exist is dropped, self-edges are dropped, duplicate source→target pairs collapse, and kind is clamped to the six permitted values. Logical relationships (identity, monitoring, dependency) are automatically drawn dashed so you can tell a network path from a conceptual one at a glance.
Layout is applied automatically when the model hasn't produced usable coordinates — if fewer than half the nodes have real positions, the diagram is laid out by tier.

The net effect is a rule that's easy to state and easy to audit: a model can suggest a relationship, but it cannot invent a resource. If a node doesn't correspond to something that genuinely exists in your tenant, it doesn't survive to the canvas.

It's a graph, not a picture

What lands is a typed model, not an image.

Edges carry meaning — six kinds, so the diagram expresses how two things relate rather than merely that they do:

Edge kind	Means	Drawn
depends_on	A needs B to function	dashed
connects_to	A talks to B	solid
data_flow	Data moves from A to B	solid
network	A network-level path (peering, subnet membership)	solid
identity	An authentication / managed-identity relationship	dashed
monitors	A observes B	dashed

Nodes are classified into twelve categories — Web & API, Compute, Containers, Databases, Storage, Integration, Networking, Security & Identity, AI & ML, Monitoring, Analytics, Other — each with its own colour, and into nine architectural layers that run edge → presentation → application → integration → data → networking → security → monitoring → shared. That layering is what lets the canvas arrange itself in tiers like a diagram a human would draw, instead of a hairball.

Groups wrap nodes by subscription, resource group, VNet or tier, which is what makes forty resources legible at all.

Reading the estate back to you

Each node renders as a card that carries rather more than a label. Alongside the name and a category-coloured Azure icon, it can show the hosting model (IaaS / PaaS / SaaS), a rough monthly cost estimate derived from type and SKU, and whether the resource is publicly or privately reachable — three questions an architect asks about every box on a whiteboard, answered on the box itself.

Nodes can also carry badges from two other systems: best-practice lint (single points of failure, missing redundancy) and assessment findings with their severity, so once you've run the Part 4 assessments the diagram becomes a heat map of where your problems physically live. Treat the cost figure as a sizing hint that tells you which corner of the diagram is expensive — not as a bill. Reconciled cost, against Cost Management, belongs to a later post.

The same diagram, six different questions

Because the diagram is a typed model rather than an image, it can be re-read without being redrawn. This is the part that surprises people in a demo, and it's where a live canvas pulls decisively ahead of a Visio file.

Azure view turns on the shared-responsibility lens: every node picks up an IaaS / PaaS / SaaS / Net badge, a globe or padlock for whether it's publicly or privately reachable, its indicative cost, and peering and private-link edges. Anything publicly reachable gains a red ring. It answers "what's exposed here?" in one toggle, on a diagram you didn't have to annotate.

WAF tint recolours the diagram by a failing Well-Architected pillar — Security, Reliability, Cost, Operations, Performance. Switch pillar and the highlight moves. Combined with the live assessment badge in the toolbar, the diagram becomes a map of where your findings physically live rather than a list of resource IDs in a report.

Boundary boxes regroup the whole diagram on demand: no boundaries, or by resource group, subscription, VNet, subnet or region. Watching the same twenty-odd resources rearrange from a flat application view into a subnet view is the fastest way to explain a network design to someone who's never seen it — and it's the clearest possible proof that this is a model, not a picture.

Hosting filter narrows to just IaaS, PaaS, SaaS or network plumbing, which is how you find the three VMs hiding in an otherwise serverless estate.

Impact is blast radius on the diagram: select a node and everything connected upstream and downstream lights up while the rest dims. Path traces the directed request path north-to-south through the system. Between them they answer the two questions every incident starts with — what does this depend on, and what breaks if it goes.

Review collects the best-practice lint into a numbered panel, and the findings also render on the nodes themselves — a Cosmos account with no private endpoint gets flagged in place, next to the resource, not in a spreadsheet.

None of these change the diagram. They're lenses over the same validated model, which is exactly why they're cheap to switch between.

A canvas you can actually work with

The diagram renders on a real graph canvas, so it behaves the way you'd expect rather than like a static export. Pan, zoom, drag nodes into a shape that matches how you think about the system, multi-select, and use the minimap when a diagram outgrows the viewport.

It's a genuine editor, not a read-only render:

Add resources by dragging from a categorised palette of Azure types.
Draw edges by dragging from one node's handle to another.
Edit anything in the inspector — a node's name, type, category, layer and metadata; an edge's label, kind and dashed/solid style; a group's name and kind.
Add groups to carve the diagram into subscriptions, resource groups or tiers.
Add sticky notes for the annotations every real diagram accumulates.

And when your manual dragging has made a mess, six layout algorithms are one click away: top-down hierarchical, left-right hierarchical, tier-layered, grid, radial and force-directed. Watching a diagram reorganise itself between them is genuinely the fastest way to find the view that makes a given system make sense.

Enhance, rebuild, and check for drift

Three buttons keep a diagram alive after the first generation.

Enhance is goal-directed AI refinement: tell it what to improve — "group these by tier", "show the data flows more clearly" — and it revises the existing diagram while preserving what's already correct, running through the same validation pass as a fresh generation.

Rebuild re-reads the live estate and regenerates. Generation runs as a background job, not a frozen browser tab: you get a phase and a percentage — scope → query → ai → save → done — so a big workload taking a couple of minutes looks like progress rather than a hang. Jobs are capped at three concurrent so a bulk rebuild can't stampede your model provider.

Drift compares the diagram against live Azure on demand — the mechanics are in the next section, but it's worth noting it's right there in the toolbar, one click from the picture it's checking.

Two more keep the diagram connected to the rest of the workbench. Ask AI puts a question box against this specific diagram, so "why is this Cosmos account reachable from the internet?" is answered with the architecture already in context. And Investigate hands the whole thing — diagram and its memory — straight into a Part 2 deep investigation, which is the loop this series has been building toward: the map you generated in ninety seconds becomes the context a war room reasons over.

Everything is undoable, and each diagram keeps an Activity log and a History of revisions, so an edit made in a hurry is never a one-way door.

Getting it back out again

A diagram trapped in a tool is a diagram that won't be used, so export is deliberately broad:

PNG and SVG for design docs, change requests and slide decks.
Mermaid to paste straight into a README or a wiki — and it round-trips: you can import an existing Mermaid flowchart back onto the canvas.
JSON for the full model, if you want to do something else with it entirely.
Bicep and Terraform skeletons — scaffolding derived from the diagram. They're a starting point, not a deployable module, and the product says so.

There's also a Present mode that takes the diagram fullscreen, which turns out to be what people actually reach for in an architecture review.

Diagrams have a lifecycle — draft → in review → ready, plus archive — and can be organised into collections, so a tenant with sixty diagrams doesn't become its own navigation problem. Every save snapshots a revision, and deletion is a recoverable Trash rather than an ending.

Honest about what it is

Two caveats the product is upfront about, and so should this post be.

The confidence value is declared by the model, not computed by the system. It's a useful signal about how sure the model was, not an independent measurement of correctness. The thing you should actually trust is the structural guarantee underneath it: every node maps to a real ARM ID, because that part is enforced in code rather than requested in a prompt.

A first pass is a first draft. Inference from properties is very good at the relationships Azure records explicitly — network paths, hosting plans, private endpoints, disks — and weaker at the ones that only exist at runtime, like an app calling an external API by URL. That's exactly why the canvas is editable and why Architecture Memory exists: the generated diagram gets you ninety percent of the way in ninety seconds, and you spend ten minutes adding what only your team knows.

Part three — Memory, drift, and the doc your on-call reads

Architecture Memory — the intent behind the diagram

A diagram tells you what is deployed. What it can't carry is what your team intended — that the second region is deliberately warm-standby rather than half-finished, that the public IP is a known and accepted risk, that when this app misbehaves you check Front Door before you touch the database. Historically that knowledge lives in one senior engineer's head and leaves when they do.

Architecture Memory is a structured document attached to each architecture, spanning 20 sections across six groups:

Group	Sections
Purpose & shape	Overview, architecture pattern, expected flow, components & responsibilities, dependencies
Topology & access	Network topology, identity & access, data & storage
Security & compliance	Security model, compliance & governance
Resilience & performance	Resiliency targets, scaling & performance, critical thresholds & SLIs
Operations	Observability, runbook / operational notes, change management, cost & sizing notes
Risk & diagnostics	Known gaps & risks, known issues & past incidents, diagnostic hints

A brand-new memory is pre-seeded with the seven highest-signal sections rather than twenty empty boxes, and the AI will draft them from the live architecture for you to correct — which is a much easier job than writing from nothing. Every save snapshots a revision (capped at 50 per document, de-duplicated by content signature), so the design history is recoverable.

Where the two pillars snap together

Here's the payoff, and it's the reason Map is worth doing before Assess.

Architecture Memory is injected into the Part 2 War Room as expert context. When a deep investigation has to fit inside a prompt budget, the memory is trimmed by an explicit priority order — expected flow first, then diagnostic hints, known gaps, critical thresholds, security model, resiliency targets, dependencies, components, network topology, observability, known issues. Eleven sections, ranked by operational value, so the most useful knowledge is the last thing to fall out of the prompt rather than the first.

Which means the note you wrote in a calm Tuesday afternoon — "check Front Door health before you look at the database" — is what a specialist agent reads at 2 a.m. while you're asleep. Documenting your architecture stops being a chore you do for auditors and becomes something that measurably improves the next incident.

Drift: when reality wanders off

Once you have a documented design and live inventory, comparing them costs almost nothing — so the workbench does it continuously.

Drift detection matches documented resources against live ones by ARM ID and sorts every resource into one of four states:

State	Meaning
ok	Present in both the design and the live estate
documented_missing	In the architecture, no longer live
live_uncontrolled	Running in your tenant, in nobody's design
inferred	An edge the architecture asserts without a live pair

It produces a drift score — simply the share of resources that reconcile — and if a workload has no architecture at all, the score is null and everything live is flagged uncontrolled, which is itself a finding worth acting on.

live_uncontrolled is usually the column people stare at. That's the resource somebody spun up in a hurry during an incident eight months ago that nobody has thought about since.

The Know-Me doc — and a refusal to make things up

Diagrams serve architects. On-call needs prose: what is this thing, who owns it, what's the SLA, who do I wake up, and what usually breaks.

The workbench generates that Know-Me doc from the architecture and its memory — 13 sections covering the workload overview, the solution architecture, the Azure services and subscriptions in scope, diagnostics and first-look triage, critical thresholds and SLIs, resiliency and DR posture, known issues and proactive callouts, security posture, support and escalation handling, contacts, and data/compliance/cost notes.

The thirteenth section is the one I'd point a sceptic at, because it's a list of everything the AI refused to invent.

The generator is explicitly forbidden from fabricating the facts that matter most: people, SLAs, RTO and RPO values, on-call rotations, customer names, subscription GUIDs, contract and schedule IDs. It doesn't guess and it doesn't produce a plausible-looking placeholder that reads like a fact. It emits a typed marker in the literal form ⟦TODO: <label> | key=<field>⟧, and every one of those markers is collected into that final checklist section for a human to complete. What Azure does know — subscriptions, regions, real scope — is auto-filled and labelled as platform-sourced, so you can see at a glance which facts came from the platform, which were AI-suggested and editable, and which are waiting on you.

Filling them in is deliberately low-friction. Fields render as the right control for their type — segmented buttons for small strict sets like Critical/High/Medium/Low, a typeahead combobox for larger sets like subscriptions or regions, a plain input for free text — and a Guided Fill mode walks you through every incomplete field one at a time, keyboard-driven, so completing a doc is a five-minute pass rather than an afternoon of hunting.

And the refusal is enforced at the door. A doc moves through draft → in review → published, and publishing is blocked until all five required groups have at least one real answer: escalation, resiliency, SLA, contract, and contacts. You cannot publish a runbook whose escalation path is still a placeholder. An on-call document that confidently states a fabricated RTO is worse than no document at all — it's a document someone will believe at 3 a.m. The gate exists so nobody discovers the gap during an incident.

Finished docs export to Markdown or a branded PDF, support inline images, keep revisions, and go to a recoverable Trash rather than vanishing.

Part four — The Estate Graph

Zoom all the way out

Per-workload diagrams answer "how does this application hang together?" The Estate Graph answers the larger question: how does all of it hang together?

It assembles the whole tenant into one interactive graph — 15 node kinds and 14 edge kinds. Nodes run from the tenant connection through management groups, subscriptions and resource groups, down to resources, and across to the artifacts you've built: workloads, architectures, memories and assessment findings. Edges carry the structure (contains, member_of, belongs_to), the modelling relationships (models, documents, has_finding) and the dependency relationships (depends_on, connects_to, data_flow).

It's workload-first by design. You navigate applications, with raw resources underneath, which is the opposite of how most inventory tools present a tenant and much closer to how teams actually think.

Five overlays, one map

Switch on an overlay and the same map answers a different question:

Overlay	Answers
Cost	Where is the money?
Retirement	What is Azure evolving away from, and by when?
Coverage	What's unmonitored, untelemetered or unprotected? (monitoring, telemetry, backup/DR)
RBAC	Who holds privileged access where?
Change	What moved recently?

That's the argument for a single graph over five separate dashboards: the cost hotspot and the coverage gap and the privileged principal are all the same node, and seeing them on one map is how you notice they're related.

Because a tenant-wide graph can get genuinely enormous, everything is capped on purpose rather than allowed to melt the browser: findings collapse into a single "⚠ N findings" super-node past 18 per workload, expanding a group loads at most 400 resources, and the overlays are bounded too (12 privileged principals, 20 retirement items, 40 change events). You choose a layout — organic, hierarchy, breadthfirst or concentric — and it persists for your tenant.

Every node is a doorway

Select a node and the inspector opens a dossier: for a workload, its environment, criticality, resource count, linked architecture, risk score and findings; for a resource, its type, SKU, location, owner, cost and the workloads it belongs to; for an architecture, its state, AI confidence and how much of its memory is filled.

From there you can trace rather than just look — expand a node's neighbours, highlight a blast radius, or ask for the path between two nodes. And the "open in" actions carry the workload context with you into Inventory, the architecture, its memory, an assessment run, Change Explorer, RBAC, Telemetry, Backup & DR — or straight into a Part 2 war-room investigation. You land somewhere useful, already scoped, instead of at the top of another dashboard.

Read-only, and why that's a design constraint rather than a limitation

Everything in this post is a read path. Workloads, architectures, memories, Know-Me docs and the Estate Graph all write to the Azure Support Agent's own storage — never to Azure. The reads themselves go straight to Azure Resource Graph and the ARM APIs.

That's a deliberate boundary. Mapping is the activity you want to be able to run against production on a Friday afternoon without a change request, without a second pair of eyes, and without thinking twice. The moment mapping can mutate, it acquires a blast radius and people stop running it — which defeats the point. Writes exist elsewhere in the workbench (applying owner tags, for instance), and they're approval-gated and audited when they do. They're a later post.

And if you haven't connected a tenant yet, demo mode seeds the whole pillar with realistic synthetic workloads, architectures and inventory, so you can walk the entire flow before pointing it at anything real.

How it's judged

Health scores are composite and explainable. Seven weighted signals, published bands, and an API that returns which signals contributed, which are missing, and the exact weights used.
Unmeasured never masquerades as bad. Signals that haven't been analyzed are excluded from the denominator rather than scored as zero, and the weights are yours to tune.
Autopilot is review-before-apply. Proposals carry reasoning and confidence, and are never persisted without an explicit human save.
Every node maps to a real resource. Model-emitted ARM IDs are validated against live inventory; duplicates collapse and orphaned edges are dropped before anything renders.
Large estates degrade gracefully. Discovery caps, property budgets and graph caps are explicit and disclosed rather than silently sampling.
The generator declines to invent. People, SLAs, RTO/RPO, contacts, GUIDs and contract IDs become explicit ⟦TODO⟧ markers, and publishing is blocked until all five required groups are genuinely complete.
Drift is measured, not asserted. Documented and live inventory reconcile by ARM ID into four named states with a score you can track over time.
Overlays reflect real data and deep links resolve. Cost, retirement, coverage, RBAC and change all read live sources, and every node opens the artifact behind it.
Read-only throughout. Mapping your estate never changes it, and the Azure Support Agent runs in your own Azure Container App against Azure's own APIs.

Try it yourself

If you've already deployed the Azure Support Agent, this is the fastest way to feel the difference — about ten minutes:

Run Autopilot against one subscription and accept a single proposed workload.
Hit Generate architecture and watch it assemble. Seeing a correct diagram of something you own build itself out of Azure Resource Graph is the moment this stops being abstract.
Open its Architecture Memory and write three honest sentences about how it's supposed to work — the expected flow, one known gap, one diagnostic hint.
Now run a Part 2 deep investigation against that workload.

The difference between the answer you get in step 4 and the answer you'd have got before step 3 is the entire argument for this pillar. The map isn't documentation you produce for someone else's benefit; it's context that makes every other answer better.

No tenant connected yet? Demo mode runs the same flow on synthetic data, with nothing to risk.

Next time — Part 4: Assess — coverage and your real posture. With the map in place, we go looking for trouble on purpose: Azure Monitor baseline alert coverage and whether the alerts you already have would actually fire, telemetry gaps, backup and DR posture with real costs attached, Well-Architected reviews at scale, FMEA risk registers, and who can do what across your estate — each judged against a versioned baseline, and each arriving with the infrastructure-as-code to close the gap.

It's open source and free to run. Three ways to start, in rising order of commitment:

Explore demo mode — synthetic data, no connection required, nothing to risk.
Deploy to your tenant — one click to Azure Container Apps; private networking; your data never leaves.

Or main repository https://github.com/zmustafa/AzureSupportAgent

Original post.

Building Resilient Power Automate Flows That Stay Running

mlotorto — Thu, 30 Jul 2026 19:04:54 GMT

If you've spent any time working with Power Automate in enterprise environments, you've seen it happen. A business-critical flow fails on a Friday afternoon. The person who built it left the company six months ago. Nobody gets an alert. Nobody knows until Monday morning when someone notices the reports didn't run. By then, the blast radius is already bigger than it needed to be.

Flow failures aren't inevitable, but unplanned ones usually share the same root causes: personal account dependencies, missing error handling, hardcoded configurations, and no visibility into what's actually happening at runtime. The good news is that most of these are solvable with patterns that aren't especially complex, they just require intentionality.

This post covers 16 recommendations across 5 focus areas that I've found to be the highest-leverage investments for making Power Automate flows more resilient, performant, and easier to operate at scale. I've ordered them roughly by the impact you'll see fastest, with supporting documentation for each section for further reading and implementation guidance.

1. Credential and Connection Resilience

This is where I always start with customers, because it's the most common source of outages and also one of the easiest to fix systematically.

Migrate high-value flows to service principal ownership. Flows owned by user accounts fail when passwords change, MFA prompts fire, or the user leaves the organization. A service principal is a non-human identity that represents an application or service and doesn't have any of those human lifecycle problems. The pattern is to register an Azure AD app registration per workload domain, grant it the required API permissions, and bind the flow to it using connection references rather than a personal connection. For connectors that don't yet support service principals (SharePoint, Outlook, Teams), use a hardened service account protected with Privileged Identity Management.

Service principal owned flows

Service Principal Licensing Requirements

Resolve broken connections

Use connection references for all connections. Embedded personal connections break when flows are exported, imported, or the owner changes. If you're building flows inside solutions, there's no good reason not to be using connection references. Create one per connector type used in your solution and bind every flow to the reference rather than a personal connection. When you deploy through environments, you update the binding once rather than hunting through individual flows.

Use connection references in solutions

Pre-populate connection references for CI/CD

Store secrets in Azure Key Vault, not flow variables. Hardcoded API keys or tokens in flow expressions show up in run history and are a real security exposure. The Key Vault connector has a "Get secret" action you can drop at the top of any flow. Restrict access to the service principal only, and when you rotate a secret in Key Vault, the flow picks it up dynamically with no changes needed.

Azure Key Vault connector reference

Key Vault secrets with environment variables

Secure data in cloud flows

2. Error Handling and Retry Logic

Unhandled errors in Power Automate produce silent failures. The flow shows red, maybe an email goes out if you're lucky, and whoever is on call has to play detective with limited context. Building proper failure boundaries before flows reach production is one of the highest-ROI things you can do.

Wrap every action in a Scope with "Configure run after." A Scope action groups a logical block of actions and lets you branch on "Has failed." The pattern is straightforward: wrap each logical unit (get data, send notification, write to Dataverse), add a parallel branch configured to run on failure, and in that branch log the error, send an alert, and terminate cleanly. This keeps failure handling isolated and predictable rather than having it bleed across the entire flow.

Use scopes to organize actions

Best practices for using scopes

Employ robust error handling

Enable built-in retry policies on connector actions. Transient HTTP and connector errors are a fact of life. The retry policy setting on any connector action handles them automatically without custom logic. Exponential interval is generally preferable over fixed. Four retries with a PT10S interval is a reasonable starting point. One important caveat: check for idempotency before enabling retries on Dataverse writes, since a write that partially succeeded before a transient error can create duplicate records if blindly retried.

Implement a retry policy

Handle transient faults (Well-Architected)

Flow limits & config (retry settings)

Use "Do until" loops with exit conditions for polling. A polling loop without bounds can run indefinitely or silently miss the exit condition it was waiting for. Set a maximum iteration count and a timeout, and make sure your exit condition is the expected success state and not just the absence of error.

Until logic documentation

Flow limits: concurrency & looping

Capture and surface error details in failure notifications. Generic failure emails like "Your flow failed" are nearly useless for triage. In the failure branch, use workflow().run.name and actions('ActionName')?['error'] expressions to pull the actual error message, and construct a direct link to the run using the environment name, flow ID, and run ID. Sending this as a Teams adaptive card with structured fields makes triage significantly faster than sifting through the admin center.

Understand error codes

3. ALM and Environment Strategy

Environment drift is sneaky. It builds up over time through ad-hoc edits made directly in production, hardcoded values that break on promotion, and manual export/import processes that are just inconsistent enough to cause problems at the worst times.

Deploy only managed solutions to production. Unmanaged solutions in production allow anyone with access to make direct edits that bypass your ALM process. The model I recommend is straightforward: build in Dev with unmanaged solutions, export as managed for promotion, validate in Sandbox, and promote to Production through Pipelines. Use the "Block unmanaged customizations" setting in Environment security policies to enforce this at the platform level.

Solution concepts with Power Platform

Create and update custom solutions for ALM in Power Platform

Use Power Platform Pipelines for automated promotion. Manual export and import is error-prone and doesn't scale. Pipelines automate the full Dev to Sandbox to Production path, support pre-deployment validation before each promotion, and let you restrict production deployment approvals to designated owners. If you're not using Pipelines yet, this is worth the setup investment.

Overview of pipelines in Power Platform

Set up pipelines in Power Platform

Deploy pipelines as a service principal

Eliminate hardcoding with Environment Variables. Flows that reference hardcoded SharePoint site URLs, list names, or IDs break every time they're promoted to a new environment. Replace them with environment variables, set the values at the environment level rather than in the solution itself, and validate bindings as part of your pre-deployment checklist.

Environment variables for Power Platform overview

Use environment variables in Power Automate solution cloud flows

Apply DLP policies scoped per environment. Overly permissive DLP in production is a real risk for data leakage and flow instability. Classify connectors intentionally across Business, Non-Business, and Blocked. Scope policies from the top down, tenant-level first and then environment-level. Use endpoint filtering to restrict rather than wholesale block where possible, and always test DLP policy changes in Sandbox before promoting them to production.

Data loss prevention policies overview

Data policies for Managed Environments

Identify blocked Power Automate flows from data policies

4. Monitoring and Observability

If you don't have visibility into flow health, you're finding out about failures the worst possible way: from users. Most organizations have significantly less monitoring in place for Power Automate than they do for their traditional application workloads.

Enable Power Platform admin center flow analytics. This is the quickest win in the monitoring category. In the Power Platform admin center, navigate to Analytics and enable flow run history and error reporting at the environment level. From there, set up a regular cadence (weekly is a good start) to review failure rates for flows classified as business-critical. Having that baseline gives you something to measure against when you start making resilience improvements.

Tenant-level analytics

Best practices for monitor your flows (analytics)

Integrate with Application Insights for deeper observability. For environments running Managed Environments, you can configure the Data Export setting in the admin center to push cloud flow runs, triggers, and actions to an Azure Application Insights resource. Once data is flowing, you get full query and visualization capability for run health, custom diagnostics, and alerting. This is the right long-term destination for any organization that takes operational reliability seriously.

Overview of integration with Application Insights

Set up Application Insights with Power Automate

5. Performance and Throttling

High-volume flows that aren't tuned for throttling will eventually hit API rate limits and start failing or queuing in ways that are hard to predict. Most of the common patterns here are straightforward to address.

Apply concurrency control on trigger-heavy flows. By default, Power Automate concurrency is unbounded. A high-volume trigger can spin up thousands of parallel runs simultaneously and hit API limits quickly. Enable concurrency control on the trigger, set the degree of parallelism based on your connector's rate limits (starting between 10 and 25 is reasonable), and monitor queue depth in the Power Automate portal to tune from there.

Understand limits & avoid throttling

Flow limits: concurrency, looping, debatching

Batch SharePoint and Dataverse operations instead of looping. An "Apply to each" loop that makes an individual API call per item scales linearly in both API call count and runtime. For SharePoint, use the "Send an HTTP request to SharePoint" action with OData $batch to group reads and writes. For Dataverse, use batch actions or FetchXML to pull multiple records in a single call. As a rough target, if a loop is making more than 50 individual API calls per run, it's a good candidate for batching.

Make batch requests with the REST APIs - SharePoint

Execute batch operations by using the Web API - Dataverse

Use chunking for large file or record sets. Flows processing large payloads can hit the 100MB message size limit or simply time out. The pattern here is to split large workloads into pages using OData $top and $skip or Dataverse paging cookies, use a parent flow to orchestrate chunked child flows, and log progress state to Dataverse so the parent can resume if a chunk fails.

Handle large messages in workflows using chunking in Azure Logic Apps (applies to Power Automate)

Message size limits in Power Automate

Where to Start

If you're looking at this list and wondering where to begin, the recommended implementation order is:

Credential and Connection Resilience first. This eliminates the most common source of outages. Migrate to service principals where supported and harden service accounts with PIM everywhere else.
Error Handling and Retry Logic second. Low effort relative to the impact. Adding scopes, run-after branches, and retry policies is mostly configuration work, and the payoff is immediate.
ALM and Environment Strategy third. Managed solutions in production, Pipelines for promotion, and environment variables in place of hardcoded values.
Monitoring and Observability fourth. Structured logging and failure alerting gives you quantifiable before-and-after visibility on everything else you're doing.
Performance and Throttling last. Address this for high-volume flows once the foundational reliability work is in place.

You don't need to implement all 16 of these at once. Even getting the first two focus areas locked down will materially reduce the failure rate for most enterprise Power Automate implementations. The goal is to build flows that run reliably, fail visibly when something goes wrong, and don't require a forensic investigation every time they do.

Start there and layer in the rest over time.

Author: Marc Lotorto Contributors: Sheldon Dsouza, Ben Kohler

Microsoft Foundry Models: A Field Guide for Cloud Solution Architects

luciacasucci — Wed, 29 Jul 2026 13:06:57 GMT

Which model families matter, what each one is for, and how to choose through the Well-Architected lens.

July 2026. Views are my own, not Microsoft guidance. Model availability, status tags, and capabilities change fast: everything below is accurate at time of writing. Verify against the live catalog before you commit an architecture.

Foundry in one minute

Microsoft Foundry is the model platform on Azure: one catalog, one control plane. What that means in practice:

1,900+ models across OpenAI, Anthropic, Mistral AI, Meta, DeepSeek, xAI, Cohere, NVIDIA, and others (at time of writing)
Two deployment modes: serverless APIs (pay per token, fastest path) and managed compute (your infrastructure, your controls)
One governance plane: Entra ID for identity, Azure billing with existing commitments applying, Azure Monitor for observability, Azure Policy for guardrails

The implication for architects: adding a model vendor is no longer a procurement cycle. It's a routing decision. That changes how you should design, and it raises the bar on the plumbing: gateway, model service layer, config-driven model names.

The lens: five pillars, five questions

The Azure Well-Architected Framework gives you five pillars. Applied to model selection, they become five questions:

Cost Optimization: does this workload need frontier pricing, or is a smaller model good enough?
Reliability: what happens when this model retires, throttles, or degrades?
Performance Efficiency: what latency does the user experience at p99, not on average?
Security: who can call this model, and what filters sit in front of it?
Operational Excellence: can you swap this model without an engineering sprint?

Each vendor section below closes with the pillar notes that matter most for that family.

[Diagram 1 – The Model Catalogue Toolbox]

OpenAI: tiers, not versions

The GPT-5 family in Foundry is not one model with version numbers. It's tiers with different jobs (at time of writing):

Deep reasoning (gpt-5.6-sol): multi-step analysis, complex agentic workflows, hard coding problems. Slowest and most expensive. Use it where being wrong is expensive.
Frontier workhorse (gpt-5.5): document work, coding, data analysis, 1M-token context. Where most complex production traffic belongs.
Chat tier (gpt-5.5 Instant): multi-turn assistants, tool orchestration, retrieval-grounded apps. Faster, cheaper, precise enough.
Nano tier (gpt-5 nano): ultra-low-latency Q&A at volume. When a user is waiting and the task is simple, this is the answer.

WAF notes

Cost + Performance: The most common failure mode is routing everything to the biggest model. Users wait longer, the bill grows, and quality doesn't improve because most requests never needed the depth.
Reliability: Every deployed model has a retirement date. Check it at design time and plan the migration path before you need it.
Operational Excellence: Abstract model names behind config. A retirement should be a config change, not a sprint.

Anthropic: Claude, now GA on Azure

Claude Opus 4.8, Claude Sonnet 5, and Claude Haiku 4.5 are generally available in Foundry hosted on Azure (GA announced June 29, 2026; Sonnet 5 followed in early July). Same Azure billing, same Entra ID, no separate Anthropic contract. Claude Fable 5, Anthropic's frontier agent model, returned to availability on July 1 after an export-control suspension from June 12-30. Mythos-class models remain gated research preview for approved organizations only (at time of writing).

Opus 4.8: maximum reasoning depth. Long-horizon agentic runs, hard multi-step analysis, complex coding.
Sonnet 5: the default for agents and coding work. Strong tool use, large context, handles workloads that used to need Opus-class models.
Haiku 4.5: the volume tier. Classification, tagging, metadata, simple Q&A at thousands of calls.

WAF notes

Security: Unlike Azure OpenAI models, Claude deployments ship with no built-in content filtering at deployment time (at time of writing). Configure Azure AI Content Safety or gateway-level filtering yourself. This is the single most-missed item in reviews.
Reliability: A safety-flagged request returns stop_reason "refusal" with HTTP 200, not an error status. Handle it in application logic, not in status-code checks.
Operational Excellence: Foundry does not pass through Anthropic's native rate-limit response headers. Use Azure Monitor for consumption tracking. And verify context-window figures in the portal's deployment pane: public documentation sources have disagreed.

Mistral AI: open weights and specialists

Mistral's lineup answers requirements the flagships don't (at time of writing):

Mistral Large 3: frontier-class capability with open weights. Tool calling, long context, agentic workflows. Deployable on managed compute when serverless doesn't fit the compliance story.
Mistral Document AI: OCR and extraction for scanned PDFs, forms, and technical manuals, feeding straight into RAG pipelines.
Codestral: purpose-built code generation.
Ministral: small models for edge and low-latency scenarios.

WAF notes

Security: Open weights plus deployment control is the answer when data residency or model inspection is a requirement, not a preference.
Cost: On document pipelines, a dedicated extraction model routinely beats prompting a general model, on both accuracy and price.

Meta Llama: the open ecosystem

Llama spans 1B and 3B small models for edge and on-device inference up to 405B for synthetic data generation and distillation. It has the widest open fine-tuning community, which matters when you need domain adaptation with full control of weights and training data.

WAF notes

Cost + Performance: Llama SLMs handle classification, extraction, and summarization at a fraction of frontier pricing, and at the edge they remove network latency entirely.
Operational Excellence: Managed-compute deployments put patching, scaling, and monitoring on your team. Budget for the ops, not just the tokens.

DeepSeek: reasoning on a budget

DeepSeek's family includes reasoning models trained for step-by-step math, science, and coding tasks at aggressive price points. Worth evaluating wherever reasoning depth matters but flagship pricing doesn't fit.

WAF notes

Cost: Benchmark against your actual reasoning workloads. The price gap to frontier reasoning tiers is large enough to justify the evaluation effort.
Security: Same rule as all partner models: bring your own content filtering, and check data-handling terms on the model card before production use.

xAI Grok and Microsoft Phi

Grok: reasoning and non-reasoning models aimed at data extraction, coding, summarization, and agentic applications.
Phi: Microsoft's small-model family, 60M+ downloads (at time of writing). On-device, edge, and high-volume simple tasks.

WAF notes

Performance: Phi at the edge removes the network round trip entirely. For latency-critical simple tasks, no hosted model competes with no network call.

The selection checklist

Before any model earns a production slot, walk it through the five pillars:

Pillar	What to check before you commit
Cost Optimization	Map the workload to a tier before picking a name. Price the small-model alternative for every high-volume task. Check reservation options for steady traffic.
Reliability	Find the retirement date before the first deployment. Plan the migration path (blue/green beats in-place for most production workloads). Know the throttling behavior and your overflow lane.
Performance Efficiency	Benchmark p95/p99 latency with your prompts, not the vendor's. Enable streaming for anything user-facing. Consider edge or SLM options where network latency dominates.
Security	Verify what content filtering ships by default; for most partner models the answer is none, so configure it. Use Managed Identities, not API keys. Check data-handling terms per model card.
Operational Excellence	Keep model names in config, never in code. Put a gateway and a model service layer in front. Define promotion gates (evals, latency budgets) before the next version ships.

The pattern that ties it together

Your most expensive model should handle your fewest requests. The hybrid pattern:

Frontier model as the brain: complex decisions, planning, orchestration. Low volume, high stakes.
Small models as the hands: Phi or Llama SLMs for classification, extraction, and summarization at volume.
Specialists where they win: reasoning models for math-heavy tasks, extraction models for documents.
A router in front: the gateway decides which request goes where. The application never knows.

The catalog isn't a leaderboard. It's a toolbox. The teams that get cost, latency, and reliability right pick per workload, not per benchmark, and they build the plumbing (gateway, model service layer, config-driven names) that makes multi-model architectures operational instead of aspirational.

[Diagram 2 – The pattern that ties the models together]

Cheat sheet: the catalog briefly

One row per family. Use it as the 30-second version of everything above (all entries at time of writing).

Family	Key models	Reach for it when	Watch out for
OpenAI GPT-5	gpt-5.6-sol, gpt-5.5, gpt-5.5 Instant, gpt-5 nano	Default frontier lineup; tiered from deep reasoning to low-latency nano	Routing everything to the flagship; retirement dates; hard-coded names
Anthropic Claude	Opus 4.8, Sonnet 5, Haiku 4.5 (GA on Azure); Fable 5	Agentic and coding workloads; second frontier vendor for resilience	No default content filtering; refusals return HTTP 200; no native rate-limit headers
Mistral AI	Large 3, Document AI, Codestral, Ministral	Open weights, data residency, document extraction pipelines	Managed compute shifts ops burden to your team
Meta Llama	1B-3B SLMs up to 405B	Fine-tuning with full weight control; edge inference; distillation	You own patching, scaling, monitoring on managed compute
DeepSeek	Reasoning model family	Step-by-step math, science, coding where flagship pricing doesn't fit	Bring your own filtering; check data-handling terms per card
xAI Grok	Reasoning + non-reasoning models	Data extraction, summarization, agentic apps	Same partner-model filtering rule
Microsoft Phi	Small-model family	On-device and edge; high-volume simple tasks	Capability ceiling; validate quality before routing volume

Disclaimer

The views expressed in this article are those of the author and do not necessarily reflect the official policy or position of Microsoft. The author is a Microsoft employee.

References

Microsoft documentation and announcements:

Vendor announcements:

Building an AI Azure Ops Workbench — Part 2: The War Room — watch AI root-cause a 5xx in 90 seconds

zmustafa — Thu, 30 Jul 2026 21:20:40 GMT

In Part 1 we made the case for the Azure Support Agent, an operations copilot that runs in your own tenant and reasons over live evidence through Azure's own APIs. This post opens the first pillar — Converse — and goes inside the two features that make it real: a chat that actually queries your estate and shows its work, and the marquee act, a War Room of specialist agents that fan out in parallel to turn "why is the site throwing 5xx?" into a ranked, validated root cause. Let's watch it work.

Every incident starts the same way: a question you can't answer fast enough. The signal you need is almost always already in Azure — the metrics are in Azure Monitor, the topology is in Azure Resource Graph, the recent changes are in the activity log, the expiring secret is in Key Vault. The hard part at 2 a.m. is that the answer is spread across a dozen surfaces and one pair of hands. The Converse pillar is about collapsing that distance: you ask in plain English, and the agent does the querying, correlating and explaining for you — against your live estate, not a training-data guess about how Azure works.

First, a chat that reads your tenant

Before we get to the marquee demo, it's worth being precise about what "chat" means here, because it is not a toy wrapped around a language model. When you ask "Why is the checkout API throwing 5xx?", the Azure Support Agent answers by doing the work — it reaches into your subscription through the official Azure MCP server (and, when you've connected it, a Microsoft Graph / Entra ID MCP server), runs read-only queries against Azure Resource Graph, Monitor and the rest, and streams the whole thing back to you as it goes.

That streaming detail matters more than it sounds. Answers arrive over Server-Sent Events, so you don't stare at a spinner — you watch a live timeline assemble: the model's reasoning as it narrates its plan, then each tool call it makes (the query it ran, how long it took, and a one-line summary of what came back), step by step, until the final answer lands. When it's done, that timeline doesn't vanish. It's persisted with the message, so if you reload the page a week later the reasoning, every tool call, and any screenshots you pasted in are all still there. Nothing about how the agent reached its conclusion is thrown away — which is exactly what you want when you have to defend that conclusion to a change board later.

A few things make it comfortable to live in day-to-day:

Multiple sessions, isolated context. Keep three investigations open at once; each chat carries its own model choice, its own Azure connection, and its own optional workload scope.
Vision input. Paste a screenshot of a failing blade or an error toast; it's part of the question and it's saved with the turn.
Starter suggestions. Not sure how to phrase it? Curated prompts, grouped by domain — connectivity, identity, security, compute, performance, cost — get you moving with one click.
Cancel-safe turns. This is the quietly brilliant part. Hit Stop, or just close the tab, and the work keeps running server-side. The turn is decoupled from your browser connection entirely: it finishes, saves its partial or full answer and its timeline, and is waiting for you when you come back. Reconnect and the stream replays what you missed and picks up live. Your investigation is never hostage to a flaky Wi-Fi connection or an accidental refresh. In Deep mode it goes one further: the investigation tree is checkpointed to the database as it builds, so even a mid-flight reconnect picks up every hypothesis already resolved.

That last point is the difference between a demo and a tool you'd actually trust mid-incident. The work is durable because the evidence is durable.

Then, when it's genuinely hard: the Azure Support Agent War Room

A single model narrating one line of investigation is perfect for "which NSG is blocking this port?" But real incidents are rarely one line of investigation — they're five plausible theories that each need chasing before you can rule them out. That's more parallel work than one reasoning thread can do well, and it's exactly where a single-threaded chatbot starts guessing.

So you flip on Deep mode, and a War Room convenes.

Instead of one generalist, the agent fields a bench of eight specialists, each with its own lens on the estate:

Specialist	What it owns
🌐 Networking	NSGs, routes, DNS, private endpoints, peering, connectivity
🔑 Identity & Access	RBAC, Entra ID, managed identities, Key Vault, secrets
⚡ Compute & Apps	VMs, App Service, AKS, Container Apps, Functions, scale sets
💾 Storage & Data	Storage accounts, SQL, Cosmos DB, disks, backups
🔐 Security & Exposure	Public exposure, Defender, open ports, permissive NSG rules
📈 Reliability & Performance	Resource health, availability, metrics, scaling, SLA
💰 Cost & Governance	Spend anomalies, Azure Policy, quotas, tags, budgets
📊 Monitoring & Logs	Activity logs, alerts, Application Insights, Log Analytics

The point of the bench isn't role-play — it's parallelism with focus. Several specialists research at the same time, each pulling live evidence for its own domain, so a broad question gets broad coverage in the time a single thread would spend on one branch. Here's what that looks like mid-flight — eight specialists working the estate at the same moment, each on its own lane:

You pick the bench — or let it pick for you

You're not stuck with all eight specialists on every question. Before a Deep investigation runs you can hand-pick the bench — just Networking and Identity for a connectivity puzzle, say — or let the agent suggest the roster for you. The suggestion isn't a black box: it proposes a recommended set with a one-line reason for each pick ("Identity — the error is a 403 and the app uses a managed identity"), and if the model is ever unsure it falls back to deterministic keyword matching, so you always get a sensible default. Pick none and it staffs the room itself.

Why bother narrowing it? Focus. A DNS question doesn't need the Cost specialist spending tool calls on spend anomalies. A tighter bench means less noise in the tree, a faster answer and lower token cost — without giving up the parallelism that makes the War Room worth it in the first place.

How an investigation actually unfolds

Under the hood the War Room runs a disciplined, four-phase play — and because it streams the whole way, you watch each phase happen rather than waiting for a black-box verdict:

Incident research. The agent spends a bounded burst of read-only tool calls building context on the live estate, then proposes a short slate of two to three distinct hypotheses, each attributed to the specialist best suited to test it. No fishing expeditions — the work is time- and iteration-boxed so it converges instead of wandering.
Hypotheses formation. The candidate root causes are pinned down as a clean set — parallel theories, not a single lucky guess — ready to be tested independently.
Validating hypotheses. Now the specialists go to work, in parallel, each running its own read-only investigation to either confirm or kill its assigned theory. Every hypothesis gets a verdict — validated, invalidated, or inconclusive — backed by a paragraph of specific evidence. A theory that survives can spawn one deeper sub-hypothesis, so the investigation drills down where the evidence leads and drops the dead ends. The result is a compact hypothesis tree (a couple of levels deep, deliberately capped so it stays readable) that records not just what was true but what was checked and ruled out.
Conclusion. With the tree resolved, the agent streams a plain-English answer — the root cause, the evidence chain that proves it, a severity, and concrete next actions — and attaches a confidence score. That score is computed, not asserted: three quarters of it comes from how the strongest hypothesis resolved and one quarter from how the whole slate resolved, so one clean validation outranks three shrugs, and an invalidated theory contributes nothing. A single validated root cause reads very differently from three inconclusive maybes, and the number says so.

Two design choices keep that four-phase play fast and honest rather than open-ended:

It's boxed, not boundless. Every phase carries both an iteration limit and a wall-clock budget, each agent's tool calls fan out only so wide, and the hypothesis tree is hard-capped in depth, breadth and total nodes. That's deliberate: an investigation that can run forever is one that runs up a bill and wanders. The budgets force it to spend its evidence-gathering where it counts and then converge — you get an answer in about a minute and a half, not a research project.

Those aren't vibes; they're constants. The guardrails, concretely:

Guardrail	Value
Hypotheses formed per investigation	2–3
Tree depth	2 — a hypothesis, plus at most one sub-hypothesis under it
Validated nodes across the whole tree	5
Research phase	≤ 5 tool-loop iterations, ~55 s wall clock
Each hypothesis validation	≤ 3 iterations, ~40 s wall clock
Concurrent tool calls inside one turn	6
Hypotheses validated in parallel	up to 12 (configurable; parallelism can be turned off entirely)

Read that table as a cost control as much as a latency one. A wide parallel search across a live tenant is exactly the kind of thing that quietly burns tokens and API quota, so the ceilings are part of the product, not a tuning afterthought.

Fast where it's cheap, strong where it counts. The broad research sweep and the parallel hypothesis validation can run on a fast, inexpensive model, while the final conclusion — the part you actually read and act on — is always written by your strongest configured model. You get the breadth of a wide parallel search without paying premium-model rates for every intermediate tool call. The fast tier is opt-in: name a cheaper provider and model for the intermediate phases and they're used automatically; leave it unset and the whole investigation runs on your primary model.

Ninety seconds after you asked, you have a conclusion rather than a hunch. Here is that exact claim, unedited, against a live tenant: one in five requests is failing with a 5xx and nothing was deployed. The room convenes, eight specialists fan out, and the verdict comes back that nothing is broken at all — the 5xx responses are being minted by the Container Apps ingress in front of an app pinned to a single replica, while the loud suspects (crash loop, autoscale churn, database exhaustion, storage) are each cleared with a number attached. And the whole thing — every hypothesis, its verdict, its evidence, the specialist who tested it — is saved as a navigable hypothesis tree attached to the chat. The next person on call doesn't start from zero; they start from your investigation.

The rest of this post is about how that happened — and there are three more recorded runs further down, including one with no incident at all.

Specialists don't just read metrics — they run real diagnostics

It's tempting to picture the bench as eight things running Resource Graph queries. They do that, but the more interesting part is that a specialist can reach for higher-order diagnostic tools mid-investigation — the same ones a senior engineer would open — all still strictly read-only:

Performance Profiler. A specialist can launch an Azure Monitor-threshold performance profile on a workload and fold the resulting scorecard and ranked bottlenecks straight into the evidence, rather than eyeballing raw metric charts.
Access and ownership lookups. "Who can actually reach this resource?" and "who owns it?" are answered from the workbench's cached RBAC and ownership scans — real least-privilege evidence, without hammering live Azure mid-incident.
In-guest sandbox checks. Where a workload has onboarded sandbox VMs and you have the permission, a specialist can run read-only in-guest diagnostics — resolve a DNS name from inside the subnet, test a connection — the checks you'd normally SSH in to run by hand.

The point is that the War Room investigates with the same toolbox you would, only in parallel and without the tab-hopping. And because Deep mode is read-only end to end, none of those tools can change a thing — a profiler runs, a VM command reads, but nothing mutates your estate.

Two investigations from the field

The single-replica case above is a clean one-liner once you see it. The real value of the War Room shows up when the obvious suspect is innocent and the truth is hiding one layer over. The next two cases are composites — teaching examples assembled from the shape of incidents I've worked, not transcripts of a single run — chosen because they show that pattern better than any one real investigation does. (If you want unedited runs, they're in the section right after this one.) In both, notice the pattern: every signal was already in Azure. The win isn't new data; it's several specialists correlating across domains at once, and a tree that records what got ruled out, not just what was true.

Investigation 1 — "The database is slow" (that isn't the database)

A checkout API starts returning intermittent 500s and stalling under peak load. On-call's gut says the SQL database is maxed out — so that's the first thing the War Room checks, and it's the first thing it rules out. The symptom points at the data layer; the cause is one layer over, and it only shows itself under load, which is exactly why a quiet repro never reproduces it.

#	Specialist	Hypothesis	Verdict	Evidence
H1	💾 Storage & Data	Azure SQL DTU/vCore saturation or blocking	Invalidated	DB sitting at ~30% DTU, no blocking waits, query stats healthy
H2	📈 Reliability & Perf	App Service plan CPU/memory saturation	Inconclusive	CPU normal — but active connection count is abnormally high
H3	🌐 Networking	SNAT port exhaustion on outbound connections from App Service	Validated	`SnatConnectionCount` hits its ceiling exactly when the 500s spike; failed-connection metric climbs under load
H3.1	🌐 Networking (sub)	App opens a fresh connection per request instead of pooling	Validated	Outbound connection churn tracks the request rate almost 1:1

The conclusion streams back in plain English: the app is exhausting its SNAT ports because it opens a new outbound connection per request instead of pooling them — so under load it simply runs out of source ports and requests start failing. The database was never the problem. Fixes follow, ranked: enable connection pooling in the app, add a NAT Gateway for outbound scale, or move SQL behind a private endpoint so those calls don't consume SNAT at all. Confidence: high — one cleanly validated chain, and the loud suspect explicitly cleared.

Here's the part worth sitting with: every number in that table lived in Azure Monitor the whole time. What was missing at 2 a.m. wasn't data — it was someone with the bandwidth to check the data, the compute and the networking layers simultaneously and notice that the SNAT ceiling, not the DB, lined up with the failures. That correlation is the whole job, and it's what the War Room does in parallel.

Investigation 2 — "Random 403s after a harmless change"

Overnight, roughly 5% of requests to an internal API start returning 403 Forbidden. It's intermittent, and it only hits some instances — the classic "works on my pod" ghost that resists every local repro. A 403 screams identity, so that's where a single-thread chatbot would dig in and stay. The War Room checks identity too — and clears it in the first pass.

#	Specialist	Hypothesis	Verdict	Evidence
H1	🔑 Identity & Access	App registration secret/cert expired	Invalidated	Credential valid for 60+ more days
H2	🔐 Security & Exposure	A new NSG / WAF rule is blocking traffic	Invalidated	No matching deny; rule set unchanged in the window
H3	📊 Monitoring & Logs	Correlate the 403 onset to a recent change	Validated (lead)	Change history shows a Private Endpoint + Private DNS zone added yesterday
H4	🌐 Networking	One spoke VNet is unlinked to the new Private DNS zone, so its instances resolve Key Vault/Storage to the public IP and hit the resource-firewall deny	Validated	The failing requests map exactly to instances in the unlinked subnet; resource-firewall logs show denied public-IP calls

The root cause: a private-endpoint rollout the day before left one spoke VNet unlinked to the new Private DNS zone. Instances in that subnet keep resolving Key Vault and Storage to their public endpoints, and the resource firewall — correctly — denies them. Everyone else is fine, which is why it looked random. The fix is a single missing VNet link. Confidence: high — a change correlated to the onset, and subnet-precise evidence tying the 403s to the gap.

This is the investigation that takes a human an hour of cross-referencing an activity log against a network diagram. In the War Room the Monitoring specialist surfaced the suspicious change while the Networking specialist proved the DNS gap — at the same time — and the tree shows the two innocent identity and security theories that got cleared along the way, so nobody re-litigates them later.

It's also a preview of where this series goes. That "intended topology: all spokes linked" is exactly the kind of fact the workbench remembers as Architecture Memory (Part 3), the missing link is the sort of thing drift detection flags before it pages anyone, and the "what changed last night" answer is Change Explorer doing in one query what used to be an hour of log archaeology (Part 5).

Four real investigations, recorded end to end

The two cases above are composites. These four are not — they are unedited screen recordings of the workbench running against a live Azure tenant, captured in one sitting. Each clip starts on an empty chat and runs the whole way through: scoping to a workload, switching the turn to Deep investigation, typing the question, the war room assembling, All Hands On Deck, the launch, the specialists working in parallel, and the written verdict at the end. Nothing is sped up beyond playback rate and nothing is cut.

You've already seen the first one — the 5xx run at the top of this post. Here's what it actually concluded, and then three more.

1. "One in five requests is failing and nothing was deployed"

The question in the title of this post, asked verbatim: roughly one in five requests to the azsupagent app fails with a 5xx while other users see no problem at all; nothing was deployed.

The verdict is the interesting part, because the answer is nothing is broken. There is no server-side fault at all — the 5xx responses are generated at the Container Apps ingress, in front of the app, because the app is pinned to a single replica (min = max = 1, zero restarts all day). Concurrent requests beyond that one instance's capacity get 502/503/504 while requests already in flight on the same replica finish normally — which is exactly the "one in five fails, the rest are fine at the same moment" shape. Crucially, the loud suspects are cleared with evidence, not skipped: no crash loop (replica count flat at 1.00, restart count 0), no autoscale churn, no Postgres exhaustion (8.88 active connections, 0 failed, CPU 8.7%), no storage dependency (100% availability across 24h).

(Recording at the top of this post.)

2. "It's slow at peak but the database dashboards look fine"

Different symptom, different specialists leading, same underlying cause as the run above — which is a good demonstration that the conclusion follows the evidence rather than the phrasing of the question. Replica count is pinned at 1.00/1.00 across the full seven-day window, so at peak the requests queue in the ingress buffer. That queueing adds seconds of latency that never appears as CPU, memory or database pressure, which is precisely why the DB dashboards look clean and why nobody can reproduce it off-peak.

Note that the database is actively disproven rather than merely unexamined: disk_queue_depth is 0 in every one of 96 buckets, peak IOPS is 4 out of a provisioned 240, and active connections peak at 15 with zero failures. That distinction — invalidated vs. not checked — is the whole argument for the hypothesis tree.

3. "What changed in the last 48 hours, and could any of it hurt me?"

This one shows the War Room used as a change-risk tool rather than an incident tool, and it found something more uncomfortable than a bad deployment. The riskiest change in the window was not made by a human at all: a service principal holding Contributor at the tenant root management group created a new, undocumented action group. The object it created is benign in content — email and SMS only, no webhook — so alerting was never broken. The identity is the finding: an unowned, unattributed non-human principal with tenant-wide write authority and root-scope policy rights.

The secondary finding is independent and equally unglamorous: shared storage account keys were retrieved three times in the small hours on an account that still permits shared-key auth, accepts TLS 1.0, and has no network restrictions at all. Meanwhile the five human container app deployments in the same window are explicitly cleared of causing instability.

4. "No incident — just tell me what breaks first"

The last one is the proactive case, and it's the one I'd actually run on a Friday afternoon. No symptom, no page, no incident: just audit this workload end to end and tell me what fails first, how bad the blast radius is, and the single highest-value fix this week.

The answer is the PostgreSQL flexible server, and the audit is specific about why: HA disabled, a single primary pinned to one availability zone, zero read replicas, geo-redundant backup disabled, seven-day retention, and storage auto-grow off. A zone outage — or an ordinary planned maintenance event — takes it fully offline with no failover target and nothing to geo-restore from, so the blast radius is a 100% workload outage with recovery bounded by manual restore time. It also flags something no single-resource check would catch: the database is the only resource in one region while the entire rest of the workload sits in another, so every data path already leaves the app's region for zero resilience benefit.

Scoped to the workload, scoped to the tenant

A War Room investigation doesn't have to boil the ocean. Point it at a Workload and it's constrained to that application's resources — its subscriptions, resource groups and resource IDs — so it reasons about the thing you care about instead of wandering the whole estate. And because the Azure Support Agent is multi-tenant aware, every investigation carries its tenant and connection context, so each tool call targets the right subscription in the right directory. One operator can run investigations across several customers' tenants from the same workbench without the signals ever crossing.

You don't have to start in the chat

Most investigations don't begin with someone typing a question — they begin somewhere else in the workbench noticing something is off. So the War Room is wired to be handed off to. When a proactive feature surfaces a risk, it can open a Deep investigation pre-loaded with the right workload and a ready-made prompt:

Backup/DR coverage finds a resource with no protection → "investigate this gap."
Change Explorer flags a risky change → "investigate this change and its blast radius."
Performance Profiler spots a metric pressing against its Azure Monitor threshold → "why is this the binding bottleneck?"
Retirement Radar finds a workload in the blast radius of an upcoming Azure retirement → "investigate this impacted workload."
Telemetry Intelligence sees a failure spike on an operation → "investigate this spike."
Architecture Memory opens the War Room with that architecture's memory already attached.
Identity findings and workload cards stage a scoped chat the same way — the workload card going straight into a deep reliability review.

Each one hands the War Room its context on a plate: deep mode on, the right workload selected, the prompt already written, the recommended bench loaded. What it deliberately does not do is start investigating. Every hand-off stops at the specialist-selection gate — nothing is dispatched until you click Launch. A dashboard can stage an investigation; only a person starts one.

There's a durable twist worth stealing, too. A browser hand-off is one-shot — it dies on a refresh — so as the War Room opens it mirrors that context into a server-side Case File, reusing the workload's already-open case (appending a note) instead of piling up duplicates. The investigation is a chat; the case is the record that outlives it.

A hand-off arrives fully staged — deep mode on, workload scoped, prompt written, bench loaded:

And then it stops, deliberately, at the gate and waits for a human:

From one workload to the whole fleet

The same machinery scales past a single incident. Select a handful of workloads in the fleet view, hit Deep review, and the workbench spins up one investigation per workload — its own chat, titled for the workload, the full eight-specialist bench, and a shared reliability brief: review this workload across networking, identity, compute, storage, security, monitoring and cost, and name the top risk. A whole portfolio gets swept for latent risk in one pass instead of you babysitting workloads one at a time.

It's throttled on purpose. A batch is capped, and the launches go out through a small concurrency gate rather than all at once — twenty simultaneous eight-specialist investigations would stampede any model provider's rate limits, and an ops tool that DDoSes your own AI quota isn't much of an ops tool. Each workload comes back with its own tree, root cause and confidence, so you can triage the fleet by what the evidence actually says. It's the difference between reacting to the workload that paged you and proactively asking all of them "is anything wrong?" — and it's the bridge to Mission Control and scheduled automations later in the series (Part 5).

It starts where you'd expect — the fleet view, with the workloads you care about selected:

What comes out: severity, confidence, and next steps

An investigation isn't just a wall of text — it resolves into a structured result you can route and reuse: a stated root cause and summary, a severity (info, warning, error or critical), the evidence chain behind the verdict, and a ranked list of next actions. Both lists are capped, so what you get is a shortlist to act on rather than a transcript to wade through. On top sits the confidence score from earlier — three parts strongest-hypothesis, one part whole-slate — so three inconclusive maybes score low and look exactly as shaky as they are. Because the shape is structured, the same result feeds a Case File, a ticket, or the next investigation without anyone re-typing it.

Those next actions can include the precise commands to fix the problem — but this is still the Converse pillar, so they stay proposal-first. The War Room tells you what it would do and why; actually doing it is the approval-gated territory of the Act pillar (Part 5). Nothing runs behind your back.

Grounded, not guessed

It's worth saying plainly why this is trustworthy, because "AI incident response" can mean a lot of hand-wavy things. Every phase here runs against live Azure evidence through MCP tools that are read-only by default — the Azure Support Agent is investigating, not changing anything. Each tool call and its result is captured in the timeline, so the conclusion is backed by a visible trail of exactly which resources were queried and what they returned. Hypotheses that the evidence contradicts are marked invalidated and kept in the tree rather than quietly dropped — the record shows the roads not taken. That read-only discipline holds even for the heavier tools: the performance profiler reads, the sandbox-VM checks read, and any write-classified tool is filtered out of the toolset the moment Deep mode starts — an investigation is structurally incapable of mutating your estate. This is the same grounding discipline the whole product is built on, and Part 6 is devoted to it in full.

It gets smarter about your estate over time

There's one more piece that turns a good investigation into an institutional one. Each architecture you've mapped can carry an Architecture Memory — the intended design, the security model, known gaps, diagnostic hints — and that context is injected into every phase of an investigation as expert background. So the War Room doesn't just know generic Azure; it knows this workload's quirks.

And the loop closes: when an investigation nails a root cause, you can save that RCA back into the architecture's Memory with a click. The next time something goes sideways on the same workload, the agent recalls what happened last time. Your incidents stop being disposable and start compounding into knowledge — the tribal wisdom that usually lives in one senior engineer's head, captured where the whole team can reuse it.

And a saved RCA isn't a dead note. It carries its confidence, it's de-duplicated against the exact investigation it came from (save it twice and you still have one entry), and it lands in the memory's known-issues section — the section that gets injected first when context has to be trimmed to fit. Your estate quietly accumulates its own case law, and the most expensively-learned facts are the last ones to fall out of the prompt. When you want a finding to leave the workbench entirely, one click escalates an investigation into a tracked Case that back-references the exact chat and message. And every Deep investigation you've run is kept in a searchable history, each with a digest — root cause, hypothesis counts, confidence — and its full, pinnable tree one click away.

The whole learning loop is one button at the bottom of a finished investigation:

…and the finding reappears in the architecture's known-issues section, ready to be injected into the next investigation on that workload:

Point it at almost anything

Root-causing a 5xx is the headline, but the same parallel-specialist machinery answers a much wider range of questions. A sampling of what people point the War Room at:

Proactive audits — "check every NSG for 0.0.0.0/0 rules" (Security & Exposure).
Change correlation — "did last night's change cause this?" (Monitoring & Logs).
Retirement impact — "which workloads does this upcoming retirement hit?" (Reliability).
Cost spikes — "why did this subscription jump 30% this month?" (Cost & Governance).
Performance — "what's the binding bottleneck on this workload?" (Reliability & Performance).
Security posture — "is any storage account publicly reachable?" (Security & Exposure).
Connectivity — "why can't this subnet reach that private endpoint?" (Networking).
Governance — "is everything tagged to our taxonomy?" (Cost & Governance).

Same four phases, same read-only discipline, same navigable tree — pointed at whatever question you have this week.

How it's judged

Every post in this series ends with the same honesty check — how you'd know the feature is actually doing its job, not just demoing well:

Every answer shows its work. Each turn carries a visible reasoning + tool-call timeline; the conclusion is traceable to the exact Azure queries behind it.
Hypotheses are validated against live evidence. The tree records what was checked, what was confirmed, and what was ruled out — each verdict tied to specific evidence.
Confidence is earned, not asserted. The score is computed from how cleanly the hypotheses resolved — weighted toward the strongest one — so a shaky conclusion looks shaky.
Read-only by default. The whole investigation runs against your estate without changing a thing; any write stays proposal-first and approval-gated.
Nothing launches itself. Dashboards can stage an investigation with full context, but specialists are only ever dispatched when a human clicks Launch.
Turns are durable. Cancel a turn or close the tab and the work continues server-side and is saved — evidence and all — so nothing is lost.
You choose the bench, or the AI suggests it. A focused roster means less noise and lower cost without losing parallel coverage.
Bounded by design. Published caps on iterations, wall-clock time, tree size and parallelism force convergence, so it lands an answer instead of wandering — and can't quietly run up a bill.
Real diagnostics, still read-only. Performance profiles, access lookups and in-guest checks all run without the power to change anything.
Reusable output. Every investigation resolves to severity, confidence and next actions — escalatable to a Case and recalled on the next incident.

Try it yourself

If you deployed the Azure Support Agent after Part 1, open a chat, ask it something real about your estate, and watch the timeline build. Then toggle Deep mode on a genuinely thorny question and let the War Room run — the hypothesis tree is the single best way to feel what an AI ops workbench does that a chatbot can't. No live tenant yet? Demo mode runs the same experience on synthetic data, with nothing to risk.

Next time — Part 3: Map your estate — from proliferation to a living picture. The War Room is only as sharp as its understanding of what's deployed, so next we turn resource proliferation into something you can reason about: Workloads with explainable health scores, Autopilot that proposes them for you, live architecture diagrams reverse-engineered from what's actually running, the Architecture Memory you just saw feeding investigations, and the whole-tenant Estate Graph. Mapping is where "proactive" gets its eyes.

It's open source and free to run. Three ways to start, in rising order of commitment:

Explore demo mode — synthetic data, no connection required, nothing to risk.
Deploy to your tenant — one click to Azure Container Apps; private networking; your data never leaves.

Or main repository https://github.com/zmustafa/AzureSupportAgent

Part 2 Original

Proactive Resiliency with Azure App Service Auto Heal

lakprasad — Wed, 22 Jul 2026 20:52:05 GMT

Modern cloud applications are expected to remain resilient - they must observe, protect, recover, and preserve service continuity when workloads begin to behave unpredictably. This includes scenarios such as transient faults, memory pressure, dependency timeouts, or sudden spikes in failed requests. While traditional monitoring detects these issues, recovery often depends on manual intervention, increasing Mean Time to Recovery (MTTR) and operational overhead.

In Azure App Service, Auto Heal is one of the platform’s bulit-in resiliency controls. It is designed to detect unhealthy application behavior and automatically take corrective action without human intervention.

This makes Auto Heal less of an operational convenience and more of an infrastructure self-preservation capability aligned with the Azure Well-Architected Framework, helping applications to recover gracefully while preserving availability and diagnostic context.

Why Auto-Heal Matters and Why Infrastructure Needs Self-Preservation

In mission-critical environments, infrastructure cannot depend entirely on humans noticing every failure pattern in real time. Applications can degrade gradually, dependencies can slow down, memory can grow unexpectedly, and individual instances can enter unhealthy states while the rest of the platform continues operating.

Without automated mitigation, these symptoms often become operational events:

A process consumes excessive memory.

Requests begin taking longer than expected.

HTTP failures increase.

One instance starts affecting user experience while others remain healthy.

Auto Heal allows App Service to respond to these conditions using pre-defined rules and it recognize that the workload has crossed a boundary where recovery action is safer than continued degradation. This way it helps maintain application stability in production environments.

How Auto-Heal Works

Trigger Conditions (Detecting Unhealthy State)

Auto-Heal monitors application behavior using configurable signals such as:

Trigger	Infrastructure signal
Request duration	The app is responding too slowly or threads may be blocked.
Request count	The app should recycle after a controlled amount of traffic.
HTTP status codes	The app is returning repeated failures.
Memory limit	The process or container is approaching unsafe resource consumption.

These conditions are evaluated over a defined time window and threshold, enabling precise detection of abnormal patterns.

Mitigation Actions (Self-Recovery)

When a trigger condition is met, Auto-Heal executes one or more corrective actions:

Recycle worker process (most common)
Restarts the application process for the affected instance.

Capture memory or crash dumps
Preserves diagnostic data for post-incident analysis.

Log events
Records Auto-Heal activity for auditing and troubleshooting.

Execute custom actions (Windows / supported Linux scenarios)
Run scripts or executables for advanced remediation.

Each action is applied per instance, ensuring that other healthy instances continue serving traffic.

Controlled Recovery Behavior

Auto-Heal works instance-by-instance, avoiding full application downtime in scaled environments.

When combined with Health Check, unhealthy instances are temporarily removed from load balancing before recovery.

A configurable startup override window prevents false triggers during application initialization.

Key Benefits:

Improved application availability
Automatically recovers from transient failures such as HTTP 5xx bursts, slow requests, or memory pressure.

Reduced Mean Time to Recovery (MTTR)
Eliminates dependency on manual restarts by triggering corrective actions instantly.

Built-in diagnostics at the point of failure
Captures logs, memory dumps, and telemetry during failure - before state is lost.

Operational efficiency
Reduces the need for continuous monitoring and manual intervention, especially during off-hours.

Seamless integration with platform capabilities
Works alongside Health Check, Azure Monitor, and Application Insights for a comprehensive resiliency strategy.

Importantly, Auto-Heal is included as part of Azure App Service and does not incur additional cost beyond standard logging and telemetry storage.

However, Auto Heal should be viewed as a guardrail, not a cure. It does not eliminate the root cause of a memory leak, dependency bottleneck, bad release, or application deadlock. Instead, it helps preserve runtime health by taking controlled action when the application reaches an unhealthy state. A resilient platform does not assume failures will never happen. It assumes failures will happen and provides mechanisms to contain, recover, and learn from them.

Hence, validate the following while configuring from a resiliency perspective:

Validation question	Why it matters
Are triggers mapped to real failure patterns?	Avoids rules that are too generic or irrelevant.
Are thresholds based on production baselines?	Prevents unnecessary recycling or delayed recovery.
Is startup time accounted for?	Avoids triggering Auto Heal during normal warm-up.
Is the action safe for the workload?	Prevents loss of in-memory state or avoidable user impact.
Are logs and diagnostics enabled?	Ensures recovery events can be investigated afterward.

Configuring Auto-Heal

Auto-Heal can be configured using multiple approaches:

Azure Portal

- Open the App Service resource in the Azure portal.

- Go to Diagnose and solve problems.

- Select Diagnostic Tools > Open Auto-Heal.

- Configure triggers, thresholds, startup delay, and mitigation actions.

Application configuration (web.config) (Windows)
Enables rule definition alongside application code

Infrastructure as Code (IaC)
ARM, Bicep, CLI, or SDK-based configuration for consistency across environments

This flexibility allows teams to standardize Auto-Heal policies across workloads.

Validating Auto-Heal in Action

Operational visibility is critical. Azure provides multiple ways to validate that Auto-Heal has triggered and executed correctly.

Azure Portal – Auto-Heal History

Navigate to:

Diagnose and Solve Problems → Auto-Heal → History

This view provides:

- Trigger conditions met

- Timestamp of execution

- Actions taken

- Status and outcome

Application Event Logs

Accessible via:

- Portal: Diagnostic Tools → Application Event Logs

- Kudu:
  LogFiles/EventLog.xml

Look for Auto-Heal related events indicating:

- Trigger activation

- Process recycles

- Success or failure of actions

Kudu Console (Advanced Diagnostics)

Kudu provides deep runtime visibility:

- Process Explorer
  Observe worker process (w3wp) restart via Process ID (PID) changes

- Log Stream
  View real-time errors leading up to Auto-Heal execution

- File system access
  Retrieve memory dumps and logs from: D:\home\LogFiles\

Review Web App Restarted & HTTP Setting Errors

- In App Service, Diagnose and Solve Problems, search for Web App Restarted, Filter by time and select Auto-Heal to view status → it will show status same as Auto Heal History

- For HTTP error scenarios, check HTTP Setting Errors for HTTPS unavailability and related triggers.

Azure Monitor Metrics

Track Auto-Heal activity using:

- WorkerProcessRecycleCount: Indicates how often instances are recycled
- HealthCheckStatus: Shows instance recovery lifecycle

These metrics can be used to:

- Build dashboards

- Configure alerts

- Correlate with performance degradation

Application Insights Integration

When enabled, Application Insights helps:

- Correlate Auto-Heal events with exceptions and dependencies

- Analyze performance trends leading to failures

- Validate recovery timelines

Best Practices for Effective Auto-Heal Usage

To maximize effectiveness:

Define safe and practical triggers

- Start with conservative thresholds to avoid unnecessary restarts.

- Continuously refine rules using diagnostics and production behavior to improve accuracy over time.

Pair Auto-Heal with platform resiliency features

- Use Health Check so traffic is routed away from unhealthy instances during recovery.

- Use autoscaling alongside Auto-Heal: Auto-Heal addresses runtime instability, while auto scale responds to load.

Monitor and learn from recovery patterns

- Track frequent events as repeated activations may indicate deeper application issues.

- Enable observability through Azure Monitor and Application Insights for end-to-end visibility.

Key Considerations and Limitations

While powerful, Auto-Heal must be used thoughtfully:

Auto Heal is not a substitute for root cause fixes
Repeated triggers indicate underlying issues that must be addressed

Overly aggressive rules can cause churn
Frequent recycles may impact user experience

Platform differences (Windows vs Linux)

- Memory-based triggers are not supported on Linux

- Custom actions have limited support on Linux

- Windows provides the most comprehensive feature set

Single-instance applications experience downtime during recycle
Scaling to multiple instances is recommended for production workloads

Dependency failures are not resolved by Auto-Heal
Recycling does not fix external service outages. The goal is not to restart applications more often. The goal is to make recovery safer, faster, and more intentional when unhealthy runtime states occur. The dependency failures need to be investigated further.

Bringing It All Together

Auto-Heal is a foundational capability for building resilient, self-recovering applications on Azure App Service. When combined with Health Check, monitoring, and scaling strategies, it enables:

Faster recovery from runtime faults

Improved availability and user experience

Reduced operational burden on engineering teams

By thoughtfully configuring triggers, validating behavior through diagnostics, and continuously refining rules, organizations can transform Auto-Heal from a simple restart mechanism into a strategic resiliency control.

Azure Support Agent — Part 1: Building an AI Azure Ops Workbench

zmustafa — Mon, 03 Aug 2026 13:11:39 GMT

Picture a busy afternoon: an app starts throwing 5xx, and you go looking for why. The good news is that Azure almost certainly already holds the answer. The Portal has the resource. Azure Resource Graph knows what's actually deployed. Monitor has the metrics, Log Analytics has the logs, Advisor has the recommendations, and the CLI reaches the rest. Azure is remarkably data-rich — very little about your estate is truly hidden. The real challenge at scale is human bandwidth: stitching a dozen surfaces into a single answer while the clock is running.

There's a second, quieter category of work, too — the things that never prompt you to open a tab in the first place. A database whose backups quietly stopped succeeding. A workload that grew business-critical before anyone gave it a DR pair. A Key Vault secret ticking toward expiry. The signals to catch every one of these already exist in Azure; what's missing is simply someone (or something) with the time to watch all of them, continuously, across the whole estate.

Now multiply that across a real environment — hundreds of resources, dozens of teams, and a platform that keeps evolving as Azure ships new capabilities and retires older ones. Nobody here is short on skill or data; there's just far more signal than any human can watch at once. And that — always-on, correlate-everything, catch-it-early work — is exactly what AI is good at. It's the gap this workbench is built to fill.

What if your subscription had a copilot that already knew the answer — and had been watching for the problem before you asked?

Meet the Azure Support Agent

The Azure Support Agent is an open-source, AI-driven operations workbench that you deploy into your own tenant. Point it at a subscription and it discovers your workloads (e.g. find related resources using AI inferencing), reverse-engineers live architecture diagrams (how difficult has it been for you to catch-up with diagrams? how about live+living documentation?), runs Well-Architected-style assessments, and puts a team of specialist AI agents at your disposal to investigate, monitor, and remediate.

It talks to Azure through the official Azure MCP server and a Microsoft Graph (Entra ID) MCP server — the same Model Context Protocol that's quickly becoming the standard way for AI to use real tools. Between them that's over a hundred read-classified tools, so the agent reasons over live evidence from your estate, not a training-data guess about how Azure works. In other words, it doesn't replace any of Azure's tooling — it orchestrates it on your behalf, on top of the platform APIs you already trust.

And it runs where your data already lives. One click deploys it to Azure Container Apps in your subscription; nothing about your estate leaves your tenant.

the Architecture Designer showing reverse-engineered architecture diagram of your workload with the help of AI, overlay with assessments, launch tools direclty

Converse — a chat that reads your tenant, and a war room for the hard ones (Part 2)

Back to that bad afternoon — except this time you don't open eight tabs. You ask, in plain English: "Why is the checkout API throwing 5xx?" — and the agent answers by doing the work, not by guessing from training data. It queries your live estate through the MCP tools and streams its reasoning and every tool call back to you as it goes, so you can see exactly which resources it looked at. Those turns are durable: reload the page and the timeline is still there, and if you cancel a long answer the work keeps running server-side instead of evaporating. You can keep several investigations going at once, paste in a screenshot of a failing blade, and lean on a starter suggestion when you're not sure what to ask.

When a question is genuinely hard, you switch on Deep mode and a War Room convenes. Instead of one model guessing, a team of specialists goes to work in parallel — a Networking specialist checks NSGs, private endpoints and DNS; an Identity specialist looks at managed identities and expiring secrets; Compute, Storage, Security and Reliability specialists each take their own angle — every one of them pulling live evidence from your estate at the same time. They form hypotheses, throw out the ones the evidence contradicts, and converge. Ninety seconds later you have a conclusion rather than a hunch: the app's Key Vault secret expired overnight; here's the dependency chain that proves it, and here's the exact command to rotate it. The whole investigation is saved as a navigable hypothesis tree attached to the chat, so the next person doesn't start from zero.

Map — turn sprawl into a picture you can actually use (Part 3)

You can't reason about resources you can't see, and almost nobody has a current map of their own estate — so the agent builds one for you. It groups raw resources into Workloads (the resources that actually make up one application) and gives each a composite, explainable health score; if you'd rather not draw those boundaries by hand, Autopilot studies the estate and proposes the groupings for you to approve.

Then it gets visual. Point it at a workload and it reverse-engineers an interactive architecture diagram from what's genuinely deployed, inferring the real relationships — NIC to subnet to VNet, app to plan, private endpoint to target — instead of trusting a stale Visio. It keeps a persistent Architecture Memory for each design (intended topology, security model, resiliency targets, known gaps) that later feeds those War Room investigations as expert context, and it flags drift when the live estate wanders from the design. It will even draft the on-call "know-me" doc for a workload, auto-filling what Azure knows and marking human-only facts — SLAs, escalation paths, RTO/RPO — as explicit TODOs rather than inventing them. Zoom all the way out and the Estate Graph renders your whole tenant as one searchable, workload-aware map, with cost, retirement and RBAC overlays and deep links straight to the underlying resource.

Use AI to discover your Azure workloads from a management group/subscription/rg scopes, then group their resources to perform proactive operations

A reverse-engineered estate graph of your Azure Workload

Assess — go looking for trouble before it finds you (Part 4)

This is where "proactive" stops being a slogan. Most tooling waits to be asked; the Azure Support Agent runs a battery of assessments that hunt for risk before the ticket exists, each judging your estate against a reviewed, versioned baseline rather than a vibe.

It starts with coverage: which resources are missing the Azure Monitor baseline alerts they ought to have, and which are shipping no diagnostic logs at all — then it hands you ready-to-apply Bicep or Terraform to close the gaps. It audits backup and DR the same way: is each resource actually backed up, with adequate retention, an offsite copy and a DR pair that's been drilled — or will you find out during the outage? A Performance Profiler reads live metrics and ranks every resource by saturation on a heatmap, naming the workload's binding bottleneck before your users feel it. And for the big picture it scores a workload against the five Well-Architected pillars with reproducible, deterministic math (never a random number a model made up), turns the architecture into a quantified FMEA risk register, and gives you a read-only identity and RBAC posture — expiring credentials, ownerless apps, MFA gaps, and who can really do what. The payoff is a change of timing: you stop discovering that a database was never backed up at the worst possible moment, and start seeing it on a dashboard on a quiet Tuesday.

Azure Monitor Baseline Alerts (AMBA) assessment against your Azure Workload - showing where you have/have not configured alerting

Performance Profiler - unified matrix view of all your resources in an Azure Workload and their performance metrics at a point in time

AI driven change explorer on what changed in your Azure Workload during a time window

Conduct AI assisted Failure Mode and Effects Analysis (FMEA) against your Azure Workload

Act — don't just find it, fix it (safely) (Part 5)

Findings that die in a dashboard help no one, so the last pillar is about closing the loop — carefully. You can dry-run an Azure Policy deny and measure its real blast radius — exactly which resources it would hit — before you enforce a thing. You can straighten out ownership and tagging from a single owner directory and tag census, applying corrections as real tags with snapshots and a one-click revert if anything looks off. And Change Explorer tells you, in plain English, what changed across your estate, who changed it, and whether it was risky — the forensics you always wish you had mid-incident.

From there it helps you get ahead of the calendar and, eventually, take yourself out of the loop. Retirement Radar maps Azure service retirements and breaking changes to the specific workloads they'll hit, with real deadlines; the Reservations view tells you whether your commitments are actually paying off or quietly leaking money. Mission Control runs every analysis on a workload at once for a single go/no-go verdict — which you can then put on a schedule with Automations, routing results to Teams, Jira or ServiceNow so the whole loop runs without you babysitting it. Everything that writes stays proposal-first: the AI suggests, you approve.

Mission Control's per-workload go/no-go verdict, pick a workload, launch and let it warm up

Unified view of Azure retirements and breaking changes against your Azure Workload

Built to be trusted

None of this is worth much if you can't hand it the keys with a clear conscience, so the trust model is deliberately conservative. The Azure Support Agent is read-only by default — every write is approval-gated and recorded in a full audit log, that can be sent to Splunk, CrowdStrike NG SIEM, or others. Full EntraID based authentication, It runs in your Azure Container App with private endpoints intra-service, so nothing about your estate ever leaves your tenant. And the AI is off until you switch it on: bring your own provider — OpenAI, Azure OpenAI, Anthropic Claude, Google Gemini, GitHub Copilot, Mistral, or a local model via Ollama or LM Studio — and there is zero LLM traffic until you explicitly configure one. This is the difference between a chatbot that answers from memory and a workbench that uses tools against live evidence, shows its working, and leaves you with artifacts you keep. It matters enough that a whole later post is devoted to it — Part 6 — covering the full access model and how the answers stay grounded, reproducible and free of fabrication.

Where this series goes

This is Part 1 of a seven-part tour. Each post ahead goes deep on one pillar of the Azure Support Agent, leads with a real screenshot, and stands on its own — so skip to whichever one solves your problem this week:

Part 2 — The War Room: watch a team of AI specialists root-cause a 5xx in ninety seconds.
Part 3 — Map your estate: live architecture diagrams, Workloads, know-me docs, the Estate Graph.
Part 4 — Assess before it pages you: alert/telemetry and backup/DR coverage, Well-Architected scores, FMEA risk registers and identity & RBAC — with the IaC to fix the gaps.
Part 5 — Act: govern, fix, and stay ahead: Policy dry-runs and blast radius, ownership & tags, change forensics, retirement radar, reservations and Mission Control.
Part 6 — Letting AI into your Azure: the read-only-by-default access model, grounded and deterministic answers, and bring-your-own-model.
Part 7 — How it's wired: the agents, the MCP layer, and one-click deploy — for the builders.

Whether you're on-call chasing a root cause, an architect who needs a current diagram, on the security team watching RBAC and secrets, or a FinOps owner hunting waste, there's a part above with your name on it.

The bottom line

The promise of the Azure Support Agent is a shift in posture. Your estate stops being a mystery you interrogate after something breaks, and becomes a system that tells you what's wrong — often before anyone gets paged.

Picture next Monday. The expiring Key Vault secret shows up on a dashboard the Thursday before, and gets rotated in the quiet — so the 5xx afternoon simply never happens. The database with the failing backups is flagged, with generated Bicep to fix it, a week ahead of the quarterly DR drill instead of during the post-mortem. Nothing about your Azure changed except when you found out.

Try it yourself

The Azure Support Agent is open source and free to run. Three ways to start, in rising order of commitment:

Explore demo mode — synthetic data, no connection required, nothing to risk.
Deploy to your tenant — one click to Azure Container Apps; private networking; your data never leaves.

Or main repository https://github.com/zmustafa/AzureSupportAgent

Next time — Part 2: The War Room — watch AI root-cause a 5xx in ninety seconds. We'll go inside a live Deep-mode investigation: how a team of specialist agents fans out across your estate in parallel, forms and discards hypotheses against real evidence, and converges on a root cause you can navigate — and trust. It's the single best demonstration of what an AI ops workbench actually feels like.

Part 1 Original

Kernels Behind the Wall: Custom Python 3.10-3.14 Jupyter Kernels for Air-Gapped Azure ML

Sabyasachi-Samaddar — Tue, 07 Jul 2026 18:28:36 GMT

Purpose

Regulated organizations - banks, insurers, healthcare providers - increasingly run Azure Machine Learning on compute instances that have no outbound internet access. Their data scientists still need modern Python versions and a rich, curated set of libraries. This document exists to close that gap. It captures a repeatable, field-tested pattern for delivering fully loaded Python 3.10-3.14 Jupyter kernels to an air-gapped Azure ML compute - without ever running a package resolver on the locked-down side. The technique itself is version-agnostic: the build, pack, ship, and unpack mechanism works for any Python version that conda can create. This guide uses Python 3.10-3.14 only as a representative range - 3.10 to match the built-in Azure ML kernel and 3.14 as today's ceiling - but the same steps apply equally to 3.9, a future 3.15, or any other version. The only version-dependent part is the package list itself (for example, the Azure ML SDK v1 tops out at Python 3.11).

What This Document Is

A hands-on runbook for building, packaging, shipping, and installing custom Python 3.10-3.14 Jupyter kernels on an air-gapped Azure ML compute instance.
A build-once, ship-an-artifact pattern that uses conda-pack, an Azure Storage blob reached over a private endpoint, and the compute's managed identity.
A cleaned, ready-to-install package list - tuned per Python version - in which the known traps from the original Azure ML package export have already been fixed (conda internals, a CLI masquerading as a library, and standard-library-shadowing backports removed; wrong package names and duplicates corrected).
A troubleshooting reference for the failures that actually happen: RBAC 403s, the azureml v1 Python ceiling, architecture and glibc mismatches, and conda-pack gotchas.

What This Document Is Not

Not an introduction to Jupyter, conda, or Python packaging fundamentals.
Not a guide to provisioning Azure ML workspaces, virtual networks, or private endpoints - those are assumed to already exist.
Not for internet-connected computes, where a normal pip install or conda environment is all you need.
Not a Windows or ARM recipe - the build box must be Linux x86_64 that matches the target compute.
Not an official Microsoft product manual; it is a practitioner's field guide.

Executive Summary

An air-gapped Azure ML compute presents two problems at the same time: it cannot reach PyPI or conda, and it ships with a single built-in Python 3.10 kernel. This guide solves both with one idea - build the environment where internet exists, freeze it into a single self-contained artifact, and ship that artifact across the air gap. The same mechanism works for any Python version conda can build - the 3.10-3.14 range used throughout is only a worked example, not a limit.

Build once on an internet-connected Linux x86_64 box, then conda-pack the environment into a single relocatable tarball.
Upload the tarball to a blob container the compute can reach over a private endpoint; on the compute, install_offline_kernel.py uses the managed identity to download and unpack it - with zero package resolution on the locked-down side.
Clean the raw package list first: strip conda internals, a CLI masquerading as a library, and backports that shadow the standard library.
Keep the azureml SDK v1 for Python 3.10 and 3.11; drop it for 3.12-3.14 and use azure-ai-ml (v2) instead.
Blob upload and download need data-plane roles (Storage Blob Data Contributor or Reader) - control-plane Owner or Contributor is not enough.
Custom kernels persist across compute stop/start; only deleting the compute removes them.

Here is a fun situation I keep running into at regulated customers. A data scientist opens a brand-new Azure ML compute instance inside a locked-down bank network, types pip install pandas, and gets… nothing. No wheels. No index. No error worth reading. Just a cursor blinking back with the quiet confidence of a machine that has never once touched the internet and does not intend to start now.

That is the environment we are dealing with: an air-gapped compute. And air-gapping is only the first of two problems. The second is that Azure ML Studio ships exactly one built-in kernel — Python 3.10. If you need 3.11, 3.12, 3.13, or 3.14, or you want that 3.10 kernel preloaded with your organisation’s curated data-science stack instead of the bare default, the only supported path is to register your own Jupyter kernel. On a normal box you’d pip your way there in a coffee break. Here you can’t, for the reason above.

This post is the pattern I use to solve both at once: build the environment somewhere that does have internet, freeze it into a single self-contained artifact with conda-pack, ship that artifact to a Storage blob the compute can reach over a private endpoint, and unpack it in place. No dependency resolution on the secure side. No firewall exceptions. No six-week change request that ends in “no.”

The short version

Two constraints, one technique. Air-gapped compute (no PyPI/conda) and a single built-in 3.10 kernel. Custom kernels solve the version/packages gap; conda-pack solves the no-internet gap.
Build once, ship a tarball. Create the env on an internet-connected Linux x86_64 box, conda-pack it, upload to blob, download + unpack on the compute. Zero resolver runs on the locked-down side.
Don’t trust the raw package list. A curated AML image’s package dump contains conda internals, a CLI masquerading as a library, and two backports that shadow the standard library. Clean it first (Section 4).
azureml SDK v1 caps you at Python 3.11. Keep azureml-* for 3.10/3.11; drop it on 3.12–3.14 and use azure-ai-ml (v2).
“Owner” does not grant blob I/O. Uploads/downloads need a data-plane role (Storage Blob Data Contributor / Reader), not control-plane Owner or Contributor.
Kernels survive stop/start. They live on the persistent user disk; only deleting the compute wipes them.

Before you start

Five things need to be true before any of the commands below will work. Confirm them now, not at Section 6:

A Linux x86_64 build box with internet — Ubuntu 22.04 or similar. Must be x86_64 (see the arch note in Section 3) and ideally a similar/older OS than the compute (see the glibc note there too).
An Azure Storage account reachable from the compute — over a private endpoint from the compute’s VNet, with a container to hold the tarball.
A managed identity on the compute instance — system- or user-assigned, attached to the compute (not just the workspace), with Storage Blob Data Reader on that account.
Rights to assign RBAC — Owner or User Access Administrator on the storage account, so you can grant the data-plane roles in Section 6. If you don’t have this, line up whoever does before you begin.
Terminal access to the compute — via AML Studio, where you’ll run the installer as azureuser.

1. Why the obvious approaches don’t work

Before the solution, it’s worth being honest about why the easy options fail, because someone on the call will suggest all of them:

The tempting shortcut	Why it doesn’t survive contact with an air-gapped compute
pip install -r requirements.txt	No route to PyPI. DNS or connect timeout. There is nothing to resolve against.
conda create -n env python=3.12 …	Can’t reach conda channels, and now also blocks on the Anaconda ToS endpoint it can’t contact.
Just use the built-in kernel	It’s Python 3.10 only, with the stock package set. No version choice, no curated stack.
“Can we open the firewall briefly?”	In a regulated tenant, that’s a security review measured in weeks and usually ends in a polite no.
Copy a wheelhouse folder by hand	Fine for pure-Python, but collapses the moment a package needs a compiled binary the target can’t build.

The insight: stop trying to make the compute reach out. Build everything outside, freeze it into one self-contained artifact — interpreter, packages, and native .so files included — and hand it to the compute through the one channel it already trusts: its own VNet’s private endpoint to Azure Storage.

2. The approach: build, pack, ship, unpack

Four steps. Assemble the env on a machine with internet, compress it with conda-pack into one .tar.gz, upload it to a blob container, and run a small installer on the compute that downloads, unpacks, and registers the kernel.

Why conda-pack instead of rebuilding from a pip freeze on the target? Because a freeze from a curated AML image is not a clean, reinstallable manifest. That env was assembled by conda plus internal feeds, so its pins are internally contradictory — they will not resolve on a fresh Python. conda-pack sidesteps all of it by shipping the exact binaries that already work. The target never runs a solver, so there is nothing left to conflict.

📌 Note

A conda-pack tarball is a frozen snapshot of a working env — same site-packages, same shared objects, same interpreter. Once conda-unpack rewrites the embedded absolute paths, it runs exactly as if it had been built on the compute. No network, no channels, no ToS prompt.

3. Part 1 — Build the environment (Linux x86_64)

Where: an internet-connected Linux x86_64 box (Ubuntu 22.04 or similar). The architecture is not optional. Build on ARM64 or a Mac and the compute rejects the binary with Exec format error. There is no cross-arch fallback; the interpreter simply won’t run.

⚠️ Gotcha

Match more than just the CPU. conda-pack ships compiled binaries, so they link against the build box’s system libraries — glibc in particular. Build on a newer OS than the compute (e.g. Ubuntu 24.04 for an older compute image) and you can hit GLIBC_2.xx not found at import time. Build on an OS that’s the same or older than the target, and match the Python patch line, to stay safe.

3.1 Install Miniconda and accept the ToS (once)

curl -fSL -o miniconda.sh https://repo.anaconda.com/miniconda/Miniconda3-latest-Linux-x86_64.sh

bash miniconda.sh -b -p $HOME/miniconda

source $HOME/miniconda/etc/profile.d/conda.sh

conda init bash && exec bash

# Accept the ToS HERE, on the box that can actually reach the endpoint:

conda tos accept --override-channels --channel https://repo.anaconda.com/pkgs/main || true

conda tos accept --override-channels --channel https://repo.anaconda.com/pkgs/r || true

conda install -n base -c conda-forge conda-pack -y

3.2 Create the env (ipykernel is mandatory)

# The env name follows through the whole workflow. Pick your version:

conda create -n aml-py310 -c conda-forge python=3.10 ipykernel pip setuptools wheel -y

conda create -n aml-py314 -c conda-forge python=3.14 ipykernel pip setuptools wheel -y

Include ipykernel. The installer on the compute runs a preflight check for it and aborts with a rebuild message if it’s missing. It comes from conda here, which is why it does not appear in the pip requirements files. Leave it out and you’ll do the whole build, upload, and download before finding out nothing registers.

3.3 Install the curated package set

A note on the requirements file: there’s no magic requirements-py310.txt attached to this post — it stands in for your own curated package list. Bring whatever set your team needs; Section 4 shows how to sanitise it and Section 5 covers the per-version differences. The one-file-per-Python-version layout below is just how I keep the versions straight.

conda activate aml-py310

pip install -r requirements/requirements-py310.txt

pip check # surface residual conflicts BEFORE you pack

# Freeze exact versions so a rebuild next month is byte-for-byte repeatable:

pip freeze > requirements/requirements-py310.lock.txt

On 3.12 / 3.13 / 3.14 this set is built around azure-ai-ml (v2); the deprecated azureml-* v1 family is intentionally absent. Section 5 explains why.

3.4 Pack it

Run this from base (where conda-pack lives), not from inside the env:

conda deactivate

mkdir -p ~/aml-offline && cd ~/aml-offline

conda pack -n aml-py310 -o aml-py310.tar.gz --ignore-missing-files

ls -lh aml-py310.tar.gz # ~150 MB minimal … several GB with the full DS stack

⚠️ Gotcha

--ignore-missing-files is required for these envs. Because the env mixes conda with a large pip layer, pip upgrades a few conda-installed packages (e.g. packaging) and rewrites their dist-info. Without the flag, conda-pack aborts with CondaPackError: Files managed by conda were found to have been deleted/overwritten. The flag packs what’s actually on disk — the consistent pip versions. Add --force if the output file already exists.

3.5 Smoke-test before shipping

mkdir -p /tmp/test && tar -xzf aml-py310.tar.gz -C /tmp/test

/tmp/test/bin/conda-unpack

/tmp/test/bin/python -c "import sys, ipykernel; print(sys.version, ipykernel.__version__)"

file /tmp/test/bin/python # MUST say: ELF 64-bit LSB executable, x86-64

rm -rf /tmp/test

Run the file check every single time. If it doesn’t say x86-64, stop and rebuild now. Shipping it anyway just moves the same failure to the compute, where it takes longer to spot.

4. Cleaning the curated package list – (Depends on the package list , Here we can see some examples)

A raw package list dumped from a curated AML image looks fine at a glance. Installed as-is, it either fails the build outright or, worse, silently breaks the env in ways you won’t notice until a notebook misbehaves three weeks later. I’ve hit six recurring problems in these lists. Here they are, with the fix for each.

Offender	The problem	The fix
conda, conda-content-trust, conda-package-handling, pycosat, boltons, ruamel-yaml-conda	These are components of the base conda install, not project libraries. pycosat needs a C toolchain, and a pip-installed conda can shadow the real one.	Remove all six. Keep ruamel.yaml (the real PyPI build) for YAML.
azure-cli	It’s a command-line app, not a notebook library, and it pins hundreds of exact azure-mgmt-* / azure-storage-* versions. Single biggest source of conflicts.	Remove it. Install the az CLI separately if you need it — don’t let it dictate your SDK versions.
azureml-core + the azureml-* v1 family	Deprecated (EOL June 2026) and version-capped. Binary deps (azureml-dataprep-rslex, dotnetcore2) only ship wheels to CPython 3.11.	Keep for 3.10/3.11 only. On 3.12–3.14, drop it and use azure-ai-ml (v2).
brotlipy	A conda-image artifact superseded by Brotli. On newer Python it often has no wheel and wants a C compiler.	Remove. requests / urllib3 pull in Brotli automatically when needed.
dataclasses==0.6, uuid==1.30	Ancient backports that shadow the standard library on modern Python. These cause real, subtle breakage — not just noise.	Remove on sight.
azure-storage-filedatalake (and duplicates)	That name 404s on PyPI — it’s the import name. azure-identity and databricks-sql-connector also appear twice.	Correct to azure-storage-file-datalake (dashes). Deduplicate the rest.

💡 Tip

Do the resolution out in the open, on the build box, where pip check and error messages are visible and cheap to fix. The whole point of shipping a tarball is that the locked-down compute never runs a solver — so make sure the solver runs cleanly before you pack.

5.The azureml v1 version ceiling

This one deserves its own section because it silently dictates which Python versions you can even offer. The azureml-* v1 SDK — azureml-core and its binary friends azureml-dataprep-rslex, azureml-dataprep-native, dotnetcore2 — is deprecated (EOL June 2026) and only ships wheels for CPython 3.8–3.11 (rslex reaches 3.12, but not 3.13/3.14). On 3.13/3.14 the install simply cannot resolve. So the strategy is version-aware:

Python	Requirements file	*azureml- v1**	Notebook SDK
3.10	requirements-py310.txt	Included	azureml-core
3.11	requirements-py311.txt	Included	azureml-core
3.12	requirements-py312.txt	Dropped	azure-ai-ml
3.13	requirements-py313.txt	Dropped	azure-ai-ml
3.14	requirements-py314.txt	Dropped	azure-ai-ml

The v2 azure-ai-ml SDK is tested on Python 3.8–3.14 and covers workspace, compute, datastore, job, pipeline, and AutoML operations — so notebooks keep working on the newer interpreters without the deprecated stack.

The filenames in this table are just a per-version naming convention for your own package lists — not files you download from anywhere. Keep one list per Python version and the rest of the workflow is identical.

6. Part 2 — Upload the tarball to blob

The tarball is packed and tested. Now drop it in a blob container the compute can reach over its private endpoint. This is where a role-assignment subtlety trips up almost everyone.

6.1 Upload options

Portal (easiest): Storage account → Containers → your container → + Add Directory offline-kernels → Upload the .tar.gz. The Portal uses the account key, so it sidesteps the RBAC issue below entirely.

azcopy with the build VM’s managed identity:

curl -fSL -o azcopy.tar.gz https://aka.ms/downloadazcopy-v10-linux

tar -xzf azcopy.tar.gz

sudo cp azcopy_linux_amd64_*/azcopy /usr/local/bin/ && sudo chmod +x /usr/local/bin/azcopy

azcopy login --identity # add --identity-client-id <uami> for user-assigned

azcopy copy "$HOME/aml-offline/aml-py310.tar.gz" \

"https://<ACCOUNT>.blob.core.windows.net/<CONTAINER>/offline-kernels/aml-py310.tar.gz"

6.2 The 403 that ignores your title

You’re a subscription Owner. Azure is unimpressed ☹. The upload still fails with 403 AuthorizationPermissionMismatch. Owner and Contributor are control-plane roles; they let you manage the storage account. Reading and writing blobs is a data-plane operation, and it needs a data-plane role the control-plane ones don’t include.

I lost most of an afternoon to this during a bank onboarding earlier this year. I was sure the role assignment just hadn’t propagated yet, so I kept re-running the copy and waiting. The real problem was that I’d been granted Contributor on the account and never the data-plane role. About ninety seconds after the right role landed, the upload went through. Check the role before you blame propagation.

Operation	Role you actually need	Plane
Upload the tarball (build side)	Storage Blob Data Contributor	Data-plane
Download the tarball (compute side)	Storage Blob Data Reader	Data-plane
Owner / Contributor	Manage the account — not sufficient for blob I/O	Control-plane

sid=$(az storage account show -n <ACCOUNT> --query id -o tsv)

az role assignment create \

--assignee <object-id-or-MI-principal-id> \

--role "Storage Blob Data Contributor" \

--scope "$sid"

# wait ~1–2 min for RBAC propagation, then retry the copy

⚠️ Gotcha

Whatever account you upload to must be the same one the installer points at, and the compute’s own managed identity needs Storage Blob Data Reader on it. Mismatch the accounts and the download 403s the same way, except now it fails on the compute instead of your build box.

7. Part 3 — Install the kernel on the compute

On the compute’s terminal (AML Studio → the compute → Terminal, where you’re azureuser), point install_offline_kernel.py at the right env and run it. The full script is in the Appendix at the end of this post.

# Constants at the top of install_offline_kernel.py:

STORAGE_ACCOUNT = "<your-account>"

CONTAINER = "data"

BLOB_PREFIX = "offline-kernels"

ENV_NAME = "aml-py310" # aml-py311 / py312 / py313 / py314

DISPLAY_NAME = "Python 3.10 (AML Custom)"

# Only for a user-assigned MI, so IMDS knows which identity to use:

export AZURE_CLIENT_ID="<uami-client-id>"

python3 install_offline_kernel.py

What the script does:

Bootstraps the SDK — installs azure-identity + azure-storage-blob into the user site (~5 MB, one-time). It also supports a local tarball or a SAS URL, so it works with zero Azure auth if you pre-stage the file.
Authenticates via DefaultAzureCredential — picks up the compute’s managed identity from IMDS.
Downloads offline-kernels/aml-py310.tar.gz over the private endpoint.
Extracts into /anaconda/envs/aml-py310 (sudo only if the path isn’t user-writable).
Runs conda-unpack — rewrites the embedded absolute paths so the env works in its new home.
Preflight-checks that the env can import ipykernel, and aborts with an exact rebuild command if not.
Registers the kernel as “Python 3.10 (AML Custom)” for azureuser. Idempotent — it skips the unpack if the env already exists.

Force a clean rebuild after re-uploading a new tarball:

sudo rm -rf /anaconda/envs/aml-py310

rm -rf ~/.local/share/jupyter/kernels/aml-py310

python3 install_offline_kernel.py

Verify: in AML Studio open a notebook → kernel picker → select “Python 3.10 (AML Custom)” and run import sys; print(sys.version). On 3.12+, reach for the v2 SDK: from azure.ai.ml import MLClient.

8. When it doesn’t work

The short list I check first, in order of how often each one is the culprit:

Symptom	Cause	Fix
Exec format error: …/bin/python	Tarball built on the wrong arch/OS (ARM64 or macOS).	Rebuild on Linux x86_64 (Section 3), re-upload, retry.
ERROR: … cannot import ipykernel	Env packed without ipykernel.	Recreate the env WITH ipykernel, repack, re-upload, clean-rebuild.
DefaultAzureCredential failed …	No MI on the compute, or MI lacks the data role.	Attach an MI to the compute (not the workspace) + Storage Blob Data Reader.
HTTP 403 from blob	MI lacks the data-plane role on this account/container.	Add Storage Blob Data Reader on the same account you uploaded to.
HTTP 404 from blob	Wrong account/container/blob path in the script.	Re-check the constants against the actual blob path in the Portal.
Multiple identities from IMDS	User-assigned MI; the SDK can’t guess which one.	export AZURE_CLIENT_ID="<uami-client-id>" first.

⚠️ Gotcha

The single most common face-plant is the workspace-vs-compute identity mix-up. Granting a role to the AML workspace identity does nothing for IMDS calls coming from the compute. The managed identity must be assigned to the compute instance itself. Check that before you check anything else.

9. Why this actually works with no internet

Four details make the whole thing hold together. They’re also exactly what a security reviewer will ask about, so it helps to have the answers ready.

IMDS is not the internet. Managed-identity auth works with zero outbound access because IMDS lives at 169.254.169.254 — a link-local address served by the Azure host, not a public endpoint. The blob download then rides the Storage account’s private endpoint inside the VNet. Nothing touches the public internet.
No Anaconda ToS prompt on the secure side. You accepted the ToS on the build box. The compute only ever sees a plain tarball, so no conda channel is ever contacted and the ToS gate never appears.
Workspace MI ≠ compute MI. RBAC granted to the workspace identity does not authorize IMDS calls from the compute. The identity must live on the compute instance. This is the top cause of a mysterious 403 — worth repeating.
Kernels persist across stop/start. The kernelspec and env live on the persistent user disk (~/.local/share/jupyter/kernels/ and /anaconda/envs/). A stop/start keeps them; only deleting the compute wipes them — at which point you just re-run the installer.

10. conda-pack gotchas on the build box

The build box has its own small collection of traps. Sharing the scars so you can skip them:

Trap	What you see	The fix
python -m conda_pack …	package cannot be directly executed	The module has no __main__. Use the console script conda-pack … or the conda pack subcommand.
conda pack	invalid choice: 'pack'	conda-pack isn’t installed in that conda. conda install -n base -c conda-forge conda-pack -y.
Packing a mixed conda+pip env	Files managed by conda … deleted/overwritten	Add --ignore-missing-files; add --force if the .tar.gz exists.
conda-pack script missing	conda-pack: command not found	python -m pip install conda-pack, then hash -r, then conda-pack -p <env-prefix> -o out.tar.gz.

Wrapping up

You don’t need to punch a hole in the firewall to give a locked-down Azure ML compute a rich, multi-version Python environment. You need to do all the messy dependency resolution outside, where you can see it, freeze the result with conda-pack, ship one tarball over a private endpoint, and register it as a kernel on the other side. Build once, pack once, upload once, install once.

The parts that bite are all predictable once you know they’re there: the arch mismatch, the azureml version ceiling, the data-plane 403, and the workspace-vs-compute identity. Now you know, so they should cost you minutes instead of an afternoon.

Appendix — install_offline_kernel.py

This is the installer referenced in Section 7 — the only script that runs on the air-gapped compute. It fetches the tarball three ways in priority order (a local file, a SAS URL, or blob download via the compute’s managed identity), unpacks it, runs conda-unpack, preflight-checks ipykernel, and registers the kernel. Set the constants at the top, copy it onto the compute, and run python3 install_offline_kernel.py.

📌 Note

Replace the placeholders. The storage account, subscription id, and CLI user below are placeholders — swap in your own values before running. Everything else is ready to use as-is.

#!/usr/bin/env python3

"""

Install an offline conda-packed Python kernel on an AML compute instance.

Gets a conda-pack tarball (in priority order):

1. a local tarball on this box (LOCAL_TARBALL, or ~/aml-offline/<ENV>.tar.gz,

or ./<ENV>.tar.gz) - no Azure auth needed,

2. a SAS URL (BLOB_SAS_URL) - no RBAC data role needed,

3. Azure Blob via DefaultAzureCredential (compute managed identity).

Then unpacks it and registers it as a Jupyter kernel for the current user.

Prerequisites (only for option 3 - blob download via managed identity):

- A managed identity (system- or user-assigned) attached to the compute instance.

- That identity has "Storage Blob Data Reader" on the storage account (DATA-plane

role; control-plane Contributor/Owner is NOT sufficient).

- For user-assigned MI, set env var AZURE_CLIENT_ID to its client id.

"""

from __future__ import annotations

import os

import subprocess

import sys

from pathlib import Path

# ---- Configuration ---------------------------------------------------------

STORAGE_ACCOUNT = "<your-account>"

CONTAINER = "data"

BLOB_PREFIX = "offline-kernels"

ENV_NAME = "aml-py310"

DISPLAY_NAME = "Python 3.10 (AML Custom)"

# If the tarball is already on this machine (e.g. you built it here), the script

# uses it and skips the blob download entirely - no Azure auth / RBAC needed.

# Leave "" to auto-search ~/aml-offline/<ENV_NAME>.tar.gz and the current dir,

# or set an explicit path.

LOCAL_TARBALL = ""

# Optional full SAS URL to the blob, e.g.

# https://<acct>.blob.core.windows.net/<cont>/<prefix>/<env>.tar.gz?sv=...

# If set, it is used instead of DefaultAzureCredential and bypasses the

# "Storage Blob Data Reader" RBAC requirement. Leave "" for managed-identity auth.

BLOB_SAS_URL = ""

# Azure auth for the blob download (only used if no local tarball and no SAS URL).

# CREDENTIAL_MODE = "cli" -> use your signed-in Azure CLI user; first run:

# az login # device-code on a compute

# az account set --subscription $SUBSCRIPTION_ID

# CREDENTIAL_MODE = "managed" -> use the compute's managed identity

# CREDENTIAL_MODE = "default" -> DefaultAzureCredential chain (MI, then CLI, ...)

SUBSCRIPTION_ID = "<your-subscription-id>"

AZURE_USERNAME = "<your-cli-user>" # expected CLI user (informational)

CREDENTIAL_MODE = "cli"

TARGET_DIR = Path(f"/anaconda/envs/{ENV_NAME}")

TARBALL_PATH = Path(f"/tmp/{ENV_NAME}.tar.gz")

# ---------------------------------------------------------------------------

def run(cmd: list[str], **kwargs) -> None:

"""Run a subprocess command, streaming output and raising on failure."""

print(f"+ {' '.join(cmd)}", flush=True)

subprocess.run(cmd, check=True, **kwargs)

def ensure_sdk_packages() -> None:

"""Install azure-identity and azure-storage-blob into the current interpreter."""

try:

import azure.identity # noqa: F401

import azure.storage.blob # noqa: F401

return

except ImportError:

pass

print("Installing azure-identity and azure-storage-blob ...", flush=True)

run([

sys.executable, "-m", "pip", "install", "--quiet", "--user",

"azure-identity", "azure-storage-blob",

])

def find_local_tarball() -> Path | None:

"""Return an existing local tarball to use instead of downloading, or None."""

candidates = []

if LOCAL_TARBALL:

candidates.append(Path(LOCAL_TARBALL).expanduser())

candidates.append(Path.home() / "aml-offline" / f"{ENV_NAME}.tar.gz")

candidates.append(Path.cwd() / f"{ENV_NAME}.tar.gz")

for c in candidates:

if c.is_file():

return c

return None

def get_credential():

"""Return an Azure credential according to CREDENTIAL_MODE."""

if CREDENTIAL_MODE == "cli":

from azure.identity import AzureCliCredential

# Uses the user signed in via `az login` (e.g. <your-cli-user>).

print(f"Using Azure CLI credential (expected user: {AZURE_USERNAME}).", flush=True)

return AzureCliCredential()

if CREDENTIAL_MODE == "managed":

from azure.identity import ManagedIdentityCredential

client_id = os.environ.get("AZURE_CLIENT_ID")

print("Using the compute's managed identity.", flush=True)

return ManagedIdentityCredential(client_id=client_id) if client_id else ManagedIdentityCredential()

from azure.identity import DefaultAzureCredential

print("Using DefaultAzureCredential chain.", flush=True)

return DefaultAzureCredential()

def download_blob(env_name: str, dest: Path) -> None:

"""Download the conda-pack tarball via SAS URL or an Azure credential."""

from azure.storage.blob import BlobClient

dest.parent.mkdir(parents=True, exist_ok=True)

if BLOB_SAS_URL:

print(f"Downloading via SAS URL -> {dest}", flush=True)

client = BlobClient.from_blob_url(BLOB_SAS_URL)

else:

ensure_sdk_packages()

credential = get_credential()

blob_name = f"{BLOB_PREFIX}/{env_name}.tar.gz"

url = f"https://{STORAGE_ACCOUNT}.blob.core.windows.net/{CONTAINER}/{blob_name}"

print(f"Downloading {url} -> {dest}", flush=True)

client = BlobClient.from_blob_url(url, credential=credential)

try:

with open(dest, "wb") as f:

stream = client.download_blob(max_concurrency=4)

stream.readinto(f)

except Exception as exc: # noqa: BLE001

msg = str(exc)

if "AuthorizationPermissionMismatch" in msg or "not authorized" in msg:

sys.exit(

"ERROR: 403 AuthorizationPermissionMismatch downloading the tarball.\n"

f"The credential in use (CREDENTIAL_MODE='{CREDENTIAL_MODE}') lacks the\n"

f"DATA-plane role 'Storage Blob Data Reader' on storage account\n"

f"'{STORAGE_ACCOUNT}'. (azcopy login / control-plane Contributor do NOT grant this.)\n"

"Fix ONE of:\n"

f" 1. Grant that role to the principal (e.g. {AZURE_USERNAME} for CLI mode,\n"

" or the compute's managed identity for managed mode), or\n"

" 2. Set BLOB_SAS_URL to a SAS link (no RBAC needed), or\n"

f" 3. Put the tarball at ~/aml-offline/{env_name}.tar.gz to skip download.\n\n"

f"Original error:\n{msg}"

)

raise

print(f"Downloaded {dest.stat().st_size:,} bytes", flush=True)

def ensure_target_dir(target: Path) -> None:

"""Create target dir, using sudo if necessary, and chown to current user."""

if target.exists():

return

user = os.environ.get("USER", "azureuser")

try:

target.mkdir(parents=True, exist_ok=True)

except PermissionError:

print(f"Need sudo to create {target}", flush=True)

run(["sudo", "mkdir", "-p", str(target)])

run(["sudo", "chown", "-R", f"{user}:{user}", str(target)])

def install_kernel() -> None:

if (TARGET_DIR / "bin").is_dir():

print(f"Env {ENV_NAME} already exists at {TARGET_DIR}, skipping unpack.", flush=True)

else:

local = find_local_tarball()

if local is not None:

print(f"Using local tarball {local}; skipping blob download.", flush=True)

src = local

downloaded = False

else:

download_blob(ENV_NAME, TARBALL_PATH)

src = TARBALL_PATH

downloaded = True

ensure_target_dir(TARGET_DIR)

print(f"Extracting {src} -> {TARGET_DIR}", flush=True)

run(["tar", "-xzf", str(src), "-C", str(TARGET_DIR)])

print("Running conda-unpack ...", flush=True)

run([str(TARGET_DIR / "bin" / "conda-unpack")])

# Only remove the temp download, never a user-provided local tarball.

if downloaded:

try:

TARBALL_PATH.unlink()

except FileNotFoundError:

pass

print(f"Registering Jupyter kernel '{ENV_NAME}' ...", flush=True)

env_python = str(TARGET_DIR / "bin" / "python")

# Sanity-check: env must actually contain ipykernel, otherwise the tarball

# was built without it and needs to be rebuilt on the Linux build box.

check = subprocess.run(

[env_python, "-c", "import ipykernel, sys; print(ipykernel.__version__)"],

capture_output=True, text=True,

)

if check.returncode != 0:

sys.exit(

f"ERROR: {env_python} cannot import ipykernel.\n"

f"The conda-pack tarball was built without ipykernel. Rebuild on the Linux box with:\n"

f" conda create -n {ENV_NAME} -c conda-forge python=<major.minor> ipykernel pip setuptools wheel -y\n"

f" conda-pack -n {ENV_NAME} -o {ENV_NAME}.tar.gz --ignore-missing-files\n"

f"then re-upload and clean up: sudo rm -rf {TARGET_DIR}\n\n"

f"stderr from check:\n{check.stderr}"

)

print(f"ipykernel {check.stdout.strip()} found in env.", flush=True)

run([

env_python, "-m", "ipykernel", "install",

"--user",

"--name", ENV_NAME,

"--display-name", DISPLAY_NAME,

])

def main() -> int:

print(f"Starting offline {ENV_NAME} kernel setup on AML compute instance ...", flush=True)

install_kernel()

kernels_dir = Path.home() / ".local" / "share" / "jupyter" / "kernels"

if kernels_dir.is_dir():

print(f"\nInstalled kernels in {kernels_dir}:", flush=True)

for k in sorted(kernels_dir.iterdir()):

print(f" - {k.name}")

print("\nOffline kernel setup completed.", flush=True)

return 0

if __name__ == "__main__":

sys.exit(main())

References

From Downtime Risk to Enterprise Resilience: SAP NetWeaver Java HA Journey with SUSE

AnuradhaKarnam — Tue, 07 Jul 2026 17:35:51 GMT

Introduction:

For years, the IT team at a global manufacturing company had relied on SAP NetWeaver 7.50 Java to support critical business processes. The environment handled everything from employee self-service applications to business integrations across multiple regions. While the infrastructure was stable, one concern remained constant: SAP Central Services instance represented a potential single point of failure.

Every maintenance window required careful planning. Any unexpected server outage could disrupt business operations, impact users worldwide, and create pressure on support teams to restore services as quickly as possible.

Challenge:

The SAP landscape was growing, and business stakeholders expected continuous availability. Traditional backup and recovery procedures could restore services, but you could not eliminate the downtime associated with infrastructure failures.

The architecture team identified SAP Java Central Services as a critical component that needed enhanced protection.

The objective was simple:

Eliminate single points of failure.
Automate failover during infrastructure issues.
Minimize business disruption.
Improve operational confidence.

Solution:

You adopted a SUSE High Availability solution built around clustered architecture. Multiple nodes were configured to work together, continuously monitoring the health of SAP services and infrastructure resources.

Using SUSE HA, the environment could automatically detect failures and transfer workloads to a healthy node whenever required. Instead of relying on manual intervention, recovery became a coordinated and automated process.

The architecture provided:

Continuous monitoring of critical SAP components.
Automated failover capabilities.
Resource management through cluster policies.
Improved service continuity during unplanned outages.

Implementation:

Deployment required close collaboration between SAP administrators, Linux engineers, and infrastructure architects. You carefully validated cluster configurations, tested failover scenarios, and documented operational procedures.

Several disaster recovery simulations were conducted:

Server shutdown tests.
Network interruption scenarios.
Service failure simulations.
Planned maintenance activities.

Each test demonstrated how the cluster could maintain application availability while reducing recovery times.

Results:

As the solution moved into production, confidence across the organization increased significantly. Business users experienced uninterrupted access to critical SAP services, while operations teams gained greater visibility into system health and cluster status.

The organization realized several key benefits:

Higher platform availability.
Faster recovery from failures.
Reduced operational risk.
Improved maintenance flexibility.
Enhanced business continuity.

Lessons Learned:

The project reinforced an important principle for enterprise SAP environments:

High availability is not simply about recovering from failure, it's about designing systems that continue operating when failures occur.
By combining SAP NetWeaver 7.50 Java Central Services with SUSE High Availability technology, the company transformed a potential business risk into a resilient and reliable platform capable of supporting mission-critical operations around the clock.

Key Benefits of SUSE HA for SAP NetWeaver Java Central Services

Elimination of Single Points of Failure
SAP NetWeaver Java Central Services (SCS) is a critical component of the SAP Java stack. SUSE High Availability (HA) protects the SCS instance by clustering resources across multiple nodes, reducing the risk of service interruption caused by server failures.
Automated Failover and Faster Recovery
When a node or service fails, SUSE Pacemaker automatically detects the issue and transfers SAP services to a healthy node. This minimizes manual intervention and significantly decreases recovery time.
Increased Business Continuity
By maintaining service availability during infrastructure outages, SUSE HA helps ensure that business-critical SAP applications remain accessible to users, reducing operational disruptions.
Reduced Downtime During Planned Maintenance
Organizations can perform maintenance activities on cluster nodes with minimal impact to SAP services. Workloads can be shifted between nodes, helping ensure uninterrupted operations.
Continuous Monitoring and Health Checks
SUSE HA continuously monitors cluster resources, SAP processes, virtual IP addresses, and storage components. Early detection of issues helps improve overall system reliability.
Improved Operational Efficiency
Automated resource management reduces the need for manual failover procedures and allows administrators to focus on proactive platform management rather than reactive troubleshooting.
Enhanced System Resilience
The cluster architecture is designed to withstand hardware, operating system, network, and service failures, creating a more robust SAP landscape.
Better Disaster Recovery Readiness
Regular failover testing and cluster validation capabilities help organizations verify their recovery procedures and improve preparedness for unexpected outages.
Enterprise-Grade Reliability for Mission-Critical Workloads
For organizations running critical SAP business processes, SUSE HA provides the foundation for achieving high availability and meeting stringent uptime requirements.

Conclusion:

In modern SAP landscapes, resilience is a business requirement rather than a technical luxury. SUSE High Availability provides organizations with a robust foundation for protecting SAP NetWeaver Java workloads, reducing downtime, and ensuring that critical business services remain available when they are needed most. SUSE High Availability for SAP NetWeaver 7.50 Java Central Services enables organizations to build a resilient SAP platform by eliminating single points of failure, automating failover, and minimizing downtime. Through continuous monitoring, intelligent resource management, and cluster-based protection, businesses can improve service continuity, simplify operations, and ensure that mission-critical SAP workloads remain available when they are needed most.

Reference links:

SUSE HA for SAP NetWeaver 7.50 Java central services | SUSE | Support Center

For the main cluster configuration and the automatic mounting of shared file systems during startup using entries defined in /etc/fstab, see the "High Availability Cluster with Simple Mount" section at the link below:

https://documentation.suse.com/sbp/sap-15/html/SAP-S4HA10-setupguide-simplemount-sle15/index.html

Refer to the resource groups outlined in the TID, together with the necessary SAP profile file modifications and location constraints documented in the "SAP NetWeaver Enqueue Replication 1 High Availability Cluster – SAP NetWeaver 7.40 and 7.50" guide linked below. These configurations are applicable since the Java SCS instance supports Enqueue Replication 1 (ENSA1) only:

https://documentation.suse.com/sbp/sap-15/html/SAP-nw740-sle15-setupguide/index.html

Azure OpenAI Architecture: The Decisions That Actually Matter (Part 3)

luciacasucci — Wed, 17 Jun 2026 16:39:08 GMT

Introduction

Part 1 of this series tackled the architectural decisions that shape any Azure OpenAI / Microsoft Foundry Models workload — capacity model, deployment scope, governance layer, grounding strategy, and quota engineering. Part 2 turned those decisions into a Well-Architected Framework discipline. Part 3 looks at the part that makes GenAI architecture genuinely different from a traditional service: the platform itself never stops moving.

Models are released, promoted to GA, moved to Legacy, deprecated, and eventually retired. New regions come online; certain features (such as Priority Processing) light up only on specific model versions and deployment scopes. Fine-tuned models inherit the lifecycle of their base. Performance characteristics shift between releases. Reliability in this world is not just uptime — it is the ability to absorb continuous change without disrupting production.

That discipline is GenAIOps: the people, processes, and tooling that turn model upgrades from emergency events into routine operations. Part 2 already covers the core lifecycle mechanics and upgrade policy trade-offs through a Well-Architected lens. Part 3 stays focused on the operational and architectural practices that make change safe: evaluation of pipelines, observability, routing patterns, prompt governance, and abstraction. Where details are time-sensitive — stage thresholds, SLA windows, regional rollout delays, capacity tier eligibility — they are flagged with "At the time of writing". Always confirm current behavior against Microsoft Learn before committing to a design.

Who is this series for?

Cloud and Solution Architects
Platform and product owners
Senior developers responsible for operating Azure OpenAI workloads in production

What you’ll learn in Part 3:

How to build an evaluation pipeline that promotes model upgrades the way CI/CD promotes code.
How to instrument full-stack observability so regressions surface early (latency, errors, token trends, quality drift).
How the Model Router pattern, canary releases, and tier-aware fallbacks turn model change into a configuration concern.
How to govern prompts as production artifacts with versioning, feature-flagged rollouts, and regression testing.
How to manage lifecycle-dependent assets (fine-tuned models) and regional rollout realities without firefighting — plus a GenAIOps Decision Matrix you can reuse as a checklist.

We’ve also included a summary decision matrix at the end of this post for quick reference.

1. Model lifecycle (recap)

Azure OpenAI/Microsoft Foundry models are living dependencies: new versions are released, promoted from Preview to GA, then eventually move through deprecation toward retirement. To avoid surprises, treat every deployed model version as having an expiration date and design so you can swap versions without rewriting application code. In general, use the Standard deployment auto-upgrade mode that preserves stability but guarantees continuity at retirement, and plan to deliberate blue/green migrations for dedicated (provisioned) capacity where auto-upgrade is not available. For the deeper mechanics (upgrade modes, retirement behavior, and migration playbooks), refer to Part 2’s Reliability section; the rest of this article focuses on the GenAIOps practices that make those upgrades routine.

Figure 1 — Models lifecycle

2. GenAIOps: Evaluating Before Promoting

Upgrading a model should not be a manual, subjective exercise. Azure AI Foundry provides evaluation capabilities that, combined with a regression prompt suite, turn model upgrades into measurable, repeatable decisions:

Side-by-side prompt comparisons across model versions.
Automated quality scoring (relevance, coherence, groundedness, safety, and fluency).
Structured-output validation (JSON conformance, schema validation).
Batch testing across comprehensive prompt libraries representative of real production traffic.
Custom evaluation metrics tailored to your domain.

Architectural best practice:

Maintain a curated regression prompt suite that mirrors real production traffic — including the long tail.
Run evaluation pipelines against candidate models before any production cut-over.
Integrate evaluation into CI/CD using Azure DevOps, GitHub Actions, or similar automation.
Define quality gates that must pass before promotion (e.g., groundedness ≥ a target threshold, p95 latency under a target budget). Pick numbers that fit your workload, not the article.

Model promotion should require passing the evaluation gates the same way application code requires passing unit tests. Without automated evaluation, model upgrades become high-risk, low-visibility events that teams avoid until forced by retirement deadlines — the exact pattern that keeps lifecycle work in the "emergency" column instead of the "scheduled" column.

Example evaluation workflow:

Trigger — a new model version reaches GA, or your migration playbook hits the R-90 step.
Deploy — the candidate model goes to a staging deployment.
Regress — the prompt suite (typically several hundred to several thousand prompts) is run against the candidate.
Compare — the candidate's outputs are scored against the current production model.
Inspect — humans review flagged differences; metrics, latency distributions, and cost-per-request go on the dashboard.
Gate — an approval step (manual or automated) decides whether the candidate proceeds to blue/green production deployment.

Figure 2 — Evaluation Pipeline

3. Observability: Full-Stack or It Didn't Happen

GenAIOps is more than one-time evaluation. Once a candidate's model has been promoted, you need continuous, end-to-end observability across the request path — not just at the model boundary. Without it, you are operating blind during model transitions.

At a minimum, instrument:

Prompt processing time (gateway through model invocation).
Model inference latency, expressed as p50, p95, and p99 — averages hide the experience of the slowest 5% of users.
Token consumption (prompt tokens, completion tokens, total) trended over time.
Error rates by class (429 throttling, 503 service unavailable, 400 validation errors, content-filter rejections).
Model version distribution — which versions are actually serving traffic right now.
User-satisfaction signals (thumbs-up/down, explicit feedback, session abandonment).

Many performance regressions only surface at scale. A model version that performs well in evaluation against a few hundred prompts may behave differently under production traffic patterns. Plan for that.

A practical metrics architecture on Azure tends to combine:

Application Insights for end-to-end request tracing across the application and gateway.
Azure Monitor for infrastructure, quota, and PTU utilization of metrics.
Custom telemetry for prompt-level success/failure tracking and quality scoring.
Log Analytics for forensic analysis when a regression is suspected.

Drift in model behavior rarely shows up as a single broken request — it surfaces as a slow shift in tail latency, fallback rate, or user-satisfaction signal. Monitoring that only looks at average will miss it.

4. The Model Router Pattern

As GenAI systems mature, a static single-model architecture becomes both limiting and expensive. A Model Router introduces dynamic, intelligent model selection in front of one or more model deployments.

Typical responsibilities of a router:

Send simple queries to a smaller, faster model and complex reasoning to a larger one.
Run canary releases of new model versions with percentage-based rollouts.
A/B test model variants to measure quality, latency, and cost differences.
Route to the right capacity tier — including falling back from Provisioned to Standard during migrations or capacity constraints.
Where the workload also needs lower-variance latency on the Standard side, route latency-critical traffic through Priority Processing on a Global Standard or Data Zone Standard (US) deployment, on a model version that supports it. (At the time of writing, Priority Processing is enabled by setting the service_tier attribute on the request and requires a model version released on or after 2025-12-01 — verify both eligibility constraints on Microsoft Learn before depending on it.

Decision logic can be driven by any combination of:

Query complexity (simple heuristics or a lightweight classifier).
User tier (e.g., free vs premium).
Response-time requirements (interactive vs background).
Cost constraints — pick the cheapest model that meets the quality bar.
Regional model availability and capacity.

Implementation options:

Azure API Management — built-in routing policies, weighted backends, retry policies.
Azure Front Door — global routing with health probes.
Custom routing service — maximum flexibility, more operational overhead.
Semantic Kernel or LangChain — framework-level routing logic embedded in the application.

Beyond cost and performance, the Model Router pattern decouples the application layer from any single model version. That decoupling is what makes lifecycle management tractable: when a model moves to Legacy, you change a router rule, not application code.

Figure 3 — Model Router Architecture vs Blue/Green Deployment

5. Prompt Lifecycle Governance

Prompts are not strings embedded in code. They are production artifacts that influence quality, cost, and safety, and they evolve almost as often as the models behind them. Treat them as first-class assets.

Prompt templates

Separate stable system instructions from dynamic content (user input and retrieved context). This lets you version, test, and audit each layer independently.

Version control

Store prompts in Git — full history, code review, branching, and tagging. Treat prompt changes the way you treat code changes: pull request, review, and test before merging.

Feature-flagged rollouts

Roll out prompt changes gradually using feature flags. Monitor the impact on a subset of users before exposing the change broadly. The same observability stack that watches model upgrades should watch prompt rollouts.

Regression testing

Maintain a regression suite of expected prompt behaviors and run it whenever prompts or models change. The suite reuses the same evaluation pipeline you built in Section 2.

Prompt-level metrics

Success rate — did the prompt achieve its intended outcome?
Fallback rate — how often did users rephrase or abandon?
Satisfaction score — explicit user feedback.
Token efficiency — average tokens per successful completion (a leading indicator of cost regression).

PII and privacy safeguards

Customer prompts and completions are not used to train base models. That means logging is safe for debugging — but defense in depth still applies:

Redact PII (names, emails, phone numbers, addresses) before logs are written.
Apply RBAC on Log Analytics workspaces so only the right roles can access raw prompt data.
Govern data retention with automated purging after a defined window.
Keep audit trails of who accessed which logs and when.

Prompt quality is not a one-time effort. It is an ongoing operational discipline that needs tooling, processing, and measurement, in the same way application code does.

Figure 4 — Prompt Lifecycle Governance

6. Fine-Tuned Models: The Hidden Retirement Risk

Fine-tuned models inherit the lifecycle of their base model. That creates a cascading retirement risk that many teams overlook.

During base-model deprecation:

New fine-tuning jobs against that base are blocked — you can no longer create new fine-tuned versions.
Existing fine-tuned deployments continue serving inference, with no immediate impact.

When the base model is retired:

Fine-tuned deployments stop responding (HTTP 404), exactly like any other deployment pinned to a retired version.

The migration imperative is straightforward: retrain fine-tuned models on the successor base model well before the retirement date, ideally during the predecessor's Legacy or Deprecated stage.

Architectural considerations:

Track base-model dependencies explicitly in your asset inventory — the same place you track library and runtime versions.
Schedule retraining workflows aligned with base-model lifecycle dates, not with team availability.
Validate fine-tuned model quality on the new base; behavior can shift between base versions.
Keep training datasets in version-controlled storage, so retraining is reproducible.
Re-evaluate whether fine-tuning is still necessary; newer base models, combined with better prompting (few-shot, chain-of-thought, structured outputs), sometimes remove the need entirely.

Common mistake: investing heavily in fine-tuning without budgeting for the recurring retraining cost and lifecycle overhead. Improved prompting on a newer base model is often the cheaper path.

7. Regional Rollouts and Multi-Region Strategy

Successor models are not always available in every Azure region simultaneously. Microsoft typically releases a new version in a subset of regions first, with broader rollout following over weeks or months. At the time of writing, the regional rollout schedule is published per model on Microsoft Learn — confirm before assuming a particular region will receive a release on a particular day.

Maintain staging deployments in early-release regions

Even if production runs elsewhere, maintain a staging deployment in regions that tend to receive new models earliest. That gives you visibility into the successor's behavior before it auto-upgrades into your primary region.

Pre-test successor models before primary auto-upgrades

If your production deployment uses "Once the current version expires", the upgrade will happen automatically. Pre-testing in an early-release region lets you catch behavioral changes before they hit live traffic.

Multi-region routing for lifecycle flexibility

Azure Front Door or Azure API Management with multi-region back-ends lets you route based on model availability, capacity headroom (one region may have quota while another is exhausted), and latency. Combined with the Model Router pattern from Section 6, this turns regional staggering from a constraint into an option.

Account for capacity-tier eligibility in your routing

Some capacity tiers are scoped to specific deployment scopes — Priority Processing, for example, is offered on Global Standard and Data Zone Standard (US) deployments at the time of writing. Bake those eligibility constraints into routing rules, so a fallback path does not silently land in an ineligible deployment.

Multi-region strategy is no longer just a disaster-recovery concern. It is also lifecycle resilience — the ability to test, stage, and absorb model changes without coupling your platform to a single region release schedule.

8. Future-Proofing Through Abstraction

Future-proofing is architectural, not procedural. The goal is to design systems that adapt to change without requiring code rewrites every time a model is promoted, deprecated, or retired.

Abstract model calls behind a service layer

Avoid calling Azure OpenAI APIs directly from the application code. Introduce an internal Model Service that owns model selection, retry and fallback, prompt-template lookup, and response validation. The application asks for an outcome ("summarize this", "classify that"); the Model Service decides which model and which prompt to use.

Externalize model names and configuration

Store model identifiers, versions, and parameters in configuration or feature flags — never as hard-coded strings. Changing models then becomes a configuration change, not a deployment.

Centralize prompt logic

Maintain prompts in a registry or template repository, not scattered across codebases. This enables centralized versioning, A/B testing without code changes, and prompt optimization that is decoupled from application releases.

Avoid scattering model identifiers across the codebase

Use constants, enums, or configuration references rather than literal model strings repeated across many files. The number of files that have to change at upgrade time is a leading indicator of how painful the upgrade will be.

Benefits of abstraction:

Seamless model replacement — swap models without touching application logic.
Multi-model strategies — the Model Router pattern becomes trivial to add.
Provider flexibility — integrating additional or alternative providers becomes a service-layer change, not an application to rewrite.
Faster adoption of new capabilities — reasoning controls, function calling, structured outputs land in one place.

Common mistake: Prototyping with direct API calls for speed and never refactoring. The technical debt accumulates until a model upgrade requires an emergency engineering sprint.

Figure 5 — Abstraction Layer for Future-Proofing.

Final Perspective

The most important shift this article asks for is a change in operational mindset:

Model upgrades are not emergencies. They are scheduled events.
Retirement deadlines are not surprising. They are published timelines, often with months of notice.
Architecture fails when teams treat models as static dependencies. They succeed when they treat models as evolving infrastructure.

In practice, GenAIOps means:

Automated evaluation that runs continuously, not just during migrations.
Controlled rollouts using blue/green or canary patterns.
Observability-driven decisions based on metrics, not intuition.
Lifecycle-aware planning, with retirement dates tracked alongside library and runtime upgrades.
Modular design that decouples applications from specific model versions.

Across the three parts of this series we have covered the architectural decisions that frame an Azure OpenAI / Microsoft Foundry Models workload (Part 1), the Well-Architected Framework discipline that keeps it sustainable (Part 2), and the GenAIOps practices that let it evolve without firefighting (Part 3). The organizations that succeed long-term are the ones that plan for model evolution from day one, invest in evaluation and observability tooling, decouple application logic from model specifics, and treat prompts and configurations as versioned artifacts.

Generative AI architecture is not about deploying a model endpoint. It is about building a platform that absorbs change gracefully as the AI landscape shifts. The retirement of a model should be a routine operational event, not a crisis. If your architecture makes model upgrades feel risky or expensive, refactor before the next retirement deadline forces your hand.

Lifecycle & GenAIOps Decision Matrix

Use this as a checklist when reviewing or signing off on the GenAIOps posture of an Azure OpenAI / Microsoft Foundry Models platform. One row per decision; one rule of thumb per row.

Area	Decision	Rule of thumb	Watch out for
Lifecycle	Version expiry tracking	Treat model versions as expiring dependencies: inventory every deployed model/version, track deprecation/retirement dates, and design so swapping versions is a configuration change (details on upgrade modes in Part 2).	Pinning versions without an owner; discovering retirement dates after an outage or emergency migration window.
Evaluation	Promotion gates	Pass the regression suite + meet domain-specific quality and latency thresholds before promoting any model.	Subjective "feels better" sign-off; gates that exist on paper but never block a release.
Evaluation	Pipeline integration	Evaluation runs in CI/CD on every candidate; the same suite watches prompt changes.	Manual evaluation runs that only happen under retirement pressure.
Observability	Latency and error metrics	Track p50/p95/p99 latency, 429/503/4xx rates, token trend, and model-version distribution. Alert on tail latency and sustained throttling.	Average-only dashboards; missed Service Health notifications for model retirements.
Observability	Quality drift	Trend per-prompt success rate, fallback rate, and user-satisfaction signals; surface drift before users complain.	Treating quality as a one-time evaluation event.
Architecture	Model Router	Centralize model selection, canary, and fallback (including Priority Processing on eligible deployments) behind a router service.	Application code that calls a specific model deployment by name; routing logic scattered across services.
Architecture	Abstraction layer	Application code asks for an outcome; the Model Service decides which model and prompt; configuration drives model selection.	Hard-coded model identifiers across many files; bypass paths that skip the service layer.
Prompts	Prompt governance	Prompts in Git, behind feature flags, with regression tests, prompt-level metrics, and PII redaction in logs.	Prompts copy-pasted across services; PII in logs; no rollback path for a regressed prompt.
Fine-tune	Fine-tuned model lifecycle	Track fine-tuned models against base-model dates; schedule retraining during the predecessor's Legacy/Deprecated window.	Treating fine-tuned models as permanent infrastructure; lost or unversioned training datasets.
Regional	Multi-region for lifecycle resilience	Maintain staging in early-release regions; route across regions to absorb staggered rollouts and capacity gaps.	Single-region production with no early-release staging; routing rules that ignore tier-eligibility constraints.

Disclaimer

I am a Microsoft employee. The views and opinions expressed in this article are my own and do not necessarily reflect those of Microsoft. This content is informational and educational; it is not an official Microsoft statement, recommendation, or commitment. Service tiers, model availability, lifecycle stages, deprecation timelines, regional rollouts, pricing, and SLAs evolve — always validate against the latest Microsoft Learn documentation before making architectural or migration decisions.

References

Power Platform tenant inventory — a community showcase of the API-first management surface

PravinT — Mon, 08 Jun 2026 13:54:14 GMT

Source: github.com/SweetsNSavories/VerseOps · MIT.

Figure 1 — VerseOps loaded against a live tenant. Per-row capacity (DB / File / Log / FinOps DB / FinOps File GB) is computed from the BAP $expand=properties.capacity call; per-env asset counts are joined client-side from the Inventory API result set. Tenant identifiers redacted; everything else is real.

Figure 2 — One environment row expanded. The row-details template fans out the inventory: Solutions / Apps / Flows / Agents (joined from the Inventory API and per-env Dataverse Web API calls), Power Pages sites (mspp_website table on the env's Dataverse), and the env's systemusers (with their assigned licenses joined from Microsoft Graph). All asset counts (9 / 3 / 53 / 241) are real.

Figure 3 — Total Assets drawer (click the Total Assets KPI tile). The whole panel is fed by a single tenant-wide Inventory API query; the per-type counts are computed client-side from assetType. The "most recent" name surfaces the freshest asset of each kind so an admin can sanity-check that the tenant feed is current.

Figure 4 — Licenses Consumed drawer (click the Licensed Users KPI tile). The list is the union of every assigned servicePlan from /users?$select=assignedLicenses rolled up to the SKU level using the tenant's subscribedSkus catalog from Microsoft Graph. SKU codes are public; the only tenant-specific data is the per-SKU seat count on the right.

Why this exists

The recurring questions at the start of every governance cycle are well known:

"How many environments do we actually have? Who owns the apps in them? How much Dataverse capacity is sitting in places no one remembers creating? Which makers left the company three months ago and still own production flows?"

The official answers — Power Platform admin center (PPAC), the Power Platform inventory page, and the Usage page — already exist and are the right starting point for daily work; they cover the common cases comprehensively. There are still moments, however, when an administrator needs:

A single offline snapshot they can search, sort, filter, and ship to a stakeholder without exposing the live admin center.
A diff between this morning and last Friday — what changed?
Joined views that the portal doesn't ship out of the box: per-env capacity × per-env asset count × per-env user count, all in one sortable grid.
The raw JSON behind every row, one click away, when something doesn't match what the portal shows.
A starting point — code they can fork, instrument, and turn into the governance tool they actually wanted.

VerseOps targets that long tail. The UI surface is roughly five files; every outbound call is enumerated in docs/network-endpoints.md. The codebase is deliberately small, read-only by design, and positioned as a foundation that adopting teams are expected to fork, instrument, and extend.

How it complements the official "Inventory" and "Usage" pages

Microsoft's Power Platform inventory gives administrators a unified view of agents, apps, and flows tenant-wide, refreshed within ~15 minutes. The Usage page tracks engagement and adoption. Both ship in the admin center today and should be every admin's first stop.

VerseOps is positioned as a complement, not a replacement:

Need	PPAC Inventory / Usage	VerseOps
Daily inventory browsing in a portal	✅ Recommended	n/a
Filter / sort / search on any column	✅	✅
Resource-detail drill-in (owner, env, dates)	✅	✅
Export to Excel	✅	✅ (CSV / cache copy)
Capacity (DB / File / Log / FinOps GB) joined per env on the same row as asset count	Partial	✅
One-click "show me the raw Dataverse / PPAC JSON" inspector	❌	✅
Local SQLite cache for offline browsing on a plane / in an air-gapped review	❌	✅
Diff between today's snapshot and yesterday's	❌	✅ (cache-based, on roadmap)
Source you can fork	n/a	✅ MIT, single solution
Telemetry sent to Microsoft / vendor	per Microsoft's data policy	None — zero outbound calls beyond Microsoft's own APIs

If you only ever need 1–4 above, stay in the admin center; it's faster and always up to date. VerseOps shows up when you need 5–11.

Architecture in one diagram

Key architectural properties:

Single process, no server-side footprint. Every call runs in the signed-in user's security context. There is no daemon, no sync job, no message bus. The operating system schedules the network calls; the user triggers a refresh.
Two distinct cloud planes. Management-plane calls (api.powerplatform.com, api.bap.microsoft.com, graph.microsoft.com) are kept separate from data-plane calls ({org}.crm.dynamics.com per environment), with audience switching handled centrally by the auth layer.
The local SQLite database is the only state. Removing %LOCALAPPDATA%\VerseOps\ returns the application to a blank slate. No other persistence exists.

What's actually feasible with the Power Platform API today

Microsoft has been very public about its shift from a UX-first to an API-first development model for Power Platform programmability: new capabilities ship in the API first, then propagate to SDKs, CLI, PowerShell cmdlets, and connectors. The Programmability and extensibility overview lays out the full toolchain — REST API, .NET SDK (Microsoft.PowerPlatform.Management), Python SDK, Power Platform CLI, PowerShell cmdlets, and the Power Platform for Admins V2 connector.

VerseOps is a deliberately small showcase of what the .NET SDK + Inventory API combination unlocks once you put a UI on it:

Capability	API used	SDK / endpoint
List every environment in the tenant with name / region / SKU / version / security group / default-flag	Power Platform API (PPAC)	Microsoft.PowerPlatform.Management SDK
Per-tenant capacity (DB / File / Log / FinOps DB / FinOps File GB)	Power Platform API (PPAC)	SDK Licensing.Tenant.GetCurrentCapacityAllocations()
Per-environment capacity in one tenant-wide call	BAP capacity (legacy GA)	GET /providers/Microsoft.BusinessAppPlatform/scopes/admin/environments?api-version=2020-10-01&$expand=properties.capacity
Every canvas app, model-driven app, code app, cloud flow, agent flow, and Copilot Studio agent in the tenant in one POST	Inventory API (preview)	POST https://api.powerplatform.com/resourcequery/resources/query?api-version=2024-10-01 (KQL-style query against PowerPlatformResources)
DLP policies + connector classification (Business / Non-Business / Blocked)	BAP Governance v2	GET /providers/PowerPlatform.Governance/v2/policies?api-version=2018-01-01
Per-env solutions / Power Pages sites / system users / roles / app + flow status	Dataverse Web API v9.2	GET {org}/api/data/v9.2/solutions, appmodules, workflows, canvasapps, systemusers, mspp_websites
User license SKU resolution + security-group display names	Microsoft Graph	GET /v1.0/subscribedSkus, /users, /groups, /directoryObjects/getByIds

The headline shape of this: one tenant-wide POST replaces what used to be N×6 per-environment GETs. For a tenant with 700 environments, that's the difference between ~4,000 round-trips per refresh and ~10. The same Microsoft.PowerPlatform.Management SDK that powers the new admin-center surfaces is the same one your tooling uses — there's no longer a "fast official one and a slow community one".

A note on the BAP API deprecation path

Several BAP routes the community has relied on for years are now in a clear consolidation track rather than a deprecation one — but the destination is the same. From the official Versioning and support page:

"The 2020-10-01 Generally available version of Power Platform API is specific to environment management and is also commonly referred to as Business Application Platform (BAP) API. The functionality of this set of endpoints are made available in the newer versions of Power Platform API along with many additional features after version 2022-03-01-preview."

In practice, what this means for tools like VerseOps:

BAP route VerseOps uses today	Status (May 2026)	Modern equivalent on api.powerplatform.com
/scopes/admin/environments?$expand=properties.capacity	GA (api-version=2020-10-01); functionally superseded but still recommended for tenant-wide capacity	Will move to a Licensing namespace endpoint as parity completes; track Programmability what's new
PowerPlatform.Governance/v2/policies (DLP)	Stable	Watch the new Connectivity / Governance namespace endpoints (e.g. List Connectors, shipped July 2025)
Microsoft.BusinessAppPlatform provider routes	All being mirrored under api.powerplatform.com namespaces (Licensing, EnvironmentManagement, AppManagement, Authorization, Governance, Connectivity)	Use the SDK — Microsoft maintains the mapping for you

Microsoft's official guidance is unambiguous: use the Power Platform API surface (api.powerplatform.com) and one of the official SDKs (.NET, Python, CLI, PowerShell, Power Platform for Admins V2 connector) for any new automation. BAP routes won't disappear without a deprecation cycle, but new features ship to api.powerplatform.com first and may never come back to BAP.

VerseOps reflects this exactly: every new feature added since April 2026 went to api.powerplatform.com, the BAP capacity client is isolated to a single ~150-line file (BapCapacityClient.cs) so it can be swapped out the moment the per-env capacity surface lands on the new API, and the token-acquisition layer (AuthService.cs) supports both audiences side by side until that day comes.

Who this helps

The MIT license permits unrestricted internal adaptation; adopting teams are encouraged to fork, re-brand, and re-sign the binary with their own enterprise code-signing certificate as part of internal distribution. Typical adopters include:

Power Platform administrators running quarterly governance reviews who need a single defensible snapshot of current tenant state.
Center-of-Excellence (CoE) leads who previously relied on the CoE Starter Kit and are moving to the in-product Inventory + Usage pages, but still require a code-level surface to extend.
FinOps and capacity owners identifying the ~5% of environments that consume 80% of Dataverse storage, with FinOps DB / FinOps File / Log GB visible on the same row as the environment name.
Mission-critical and regulated workloads (financial services, healthcare, public sector) where a desktop tool that authenticates as the human administrator, emits zero telemetry, and stores all state locally is materially easier to risk-accept than a SaaS dashboard.
Security and penetration-test teams who require a reproducible, auditable, signed Windows binary and a clear inventory of what it touches. The SBOM, SECURITY.md, SIGNING.md, and CodeQL workflow are committed to the repository.
Engineering teams learning the Power Platform API who want a non-trivial, well-commented .NET sample that exercises every major namespace.

Where this could go next

The same API surface that powers VerseOps today can support a substantially richer set of experiences. Candidate directions follow; community input on prioritization is welcome via the issue tracker.

1. An agentic governance assistant

Wrap the local SQLite cache + the same auth pipeline behind a Microsoft 365 Copilot agent (or a Foundry agent), and let an admin ask things like:

"Which environments grew the most this week and who owns the new flows?"
"List every canvas app with a deprecated connector that's still 'On' in a production env."
"Show me orphaned resources owned by users disabled in Entra in the last 30 days."

The Power Platform API + Inventory API already returns everything you need to answer these in seconds. The agent surface is just a new face for the same data — and because the cache is local, the agent can run without ever sending tenant data to a third party.

2. Periodic snapshots → drift report

A scheduled task that runs VerseOps.App --refresh --headless once a day, writes the SQLite snapshot to a versioned folder, and emails a delta. "Today vs yesterday: +12 canvas apps in the Default env, –3 envs decommissioned, capacity climbed 4.1 GB on org-prod-eu."

3. Multi-tenant fan-out for MSPs / consultancies

Same EXE, multiple tenant profiles, side-by-side comparison view. The auth layer already supports --tenant <guid>; the cache schema is per-tenant-keyed.

4. Plug-ins for the Inventory API custom queries

The Inventory API's POST /resourcequery/resources/query accepts arbitrary KQL-style projections. A plug-in directory of "common admin questions as queries" (orphaned apps, oldest unused flows, premium connector usage by env) could grow organically.

5. Sister tools in Python / TypeScript

The Python SDK is GA; a Jupyter notebook that mirrors VerseOps' three core panels (env list + capacity + assets) would be ~200 lines and would land instantly with the data-science crowd.

The repository is intended as a working base for these explorations: the foundational ~80% — authentication, caching, paging, retry, redaction, error capture, and theming — is already implemented and exercised against a live ~700-environment tenant. Proposals for any of the directions above can be filed on the issue tracker.

What's in the repository

Everything below is on main at github.com/SweetsNSavories/VerseOps, MIT-licensed:

The single WPF EXE — VerseOps.App/
API clients, one per Microsoft service — VerseOps.App/Inventory/Services/
SQLite catalog schema — VerseOps.App/Inventory/Sql/schema.sql
README.md — install, run, build
SECURITY.md — disclosure policy + threat model
SIGNING.md — three publish-with-signature paths (self-signed dev, Azure Trusted Signing, OV/EV)
docs/network-endpoints.md — every outbound host + OAuth scope
THIRD-PARTY-NOTICES.md + sbom.cdx.json — full dependency attribution + CycloneDX SBOM
CI: build, vulnerability scan, CodeQL — .github/workflows/
Branch protection ruleset (PR required, force-push blocked) — .github/branch-protection.json

Try it

git clone https://github.com/SweetsNSavories/VerseOps.git cd VerseOps dotnet build VerseOps.sln -c Release .\VerseOps.App\bin\Release\net10.0-windows\VerseOps.App.exe

Sign in with a tenant admin account (Power Platform Administrator or Dynamics 365 Administrator), click Refresh, and the first cold pull populates the local cache. Subsequent launches are instant from the cache; click Refresh again whenever you want a fresh snapshot.

Closing thought

The thesis behind this post is straightforward: an API-first Power Platform management surface puts a complete tenant inventory within reach in roughly 3,000 lines of C#. The official Inventory and Usage pages remain the right tool for daily-driver scenarios. The SDK and Inventory API together cover the long tail — the cases where an organization needs a specific join, requires offline operation, or needs the answer the same week.

VerseOps is offered to the community as that starting point. Issues and pull requests are welcome on the public tracker; security disclosures should follow the process documented in SECURITY.md.

— Praveen T · maintainer, VerseOps

References

Power Platform inventory — the in-product surface VerseOps complements
Power Platform admin center Usage page
Programmability and extensibility overview — official tooling map
Versioning and support — the BAP-vs-PPAC story
Programmability — What's new or changed — monthly release log
Power Platform API REST reference (latest)
Microsoft.PowerPlatform.Management on NuGet — the .NET SDK VerseOps consumes
Power Platform for Admins V2 connector — the no-code path to the same API
Tutorial: Create a daily capacity report — Microsoft's own end-to-end SDK example
Power Platform URLs and IP address ranges — for network allow-lists

Archiving Years of Dataverse Audit History Before You Prune It — A Pragmatic, Open Source Pattern

PravinT — Mon, 08 Jun 2026 13:53:50 GMT

Why the audit table is special

The audit table is different from the rest of Dataverse in two ways that matter for an archive design:

It’s an immutable, append-only record of who changed what, when, and from where — the closest thing Dataverse has to a forensic ledger.
The valuable part of an audit row is not the row itself; it’s the diff (old value → new value, attribute mask, related-record context). The audit row stores that diff in a packed changedata column, and the bound RetrieveAuditDetails function is what decodes it into a structured OldValue / NewValue / ChangedAttributes shape your downstream tools can actually query. Synapse Link with the Delta Lake profile will carry the changedata column to the lake, but you still need a parser on the other side; this pattern calls RetrieveAuditDetails at archive-time so what lands in the destination is already decoded and immediately queryable.

That combination makes the audit table the single most useful Dataverse table for:

Regulatory and compliance investigations
“Why did this opportunity status change in Q3 of 2022?” forensic queries (years after the fact)
Internal analytics on user behaviour and process adoption

It’s also the table that grows the fastest. The Dataverse default retention is 90 days, but in practice many enterprises extend that to several years — or set it to never delete — to retain evidence for compliance and forensic review. The result, often after five to seven years, is an audit table holding tens of GB to multiple TB of capacity, dominating the entitlement bill, and rarely accessed in normal operations.

At that point the storage conversation becomes unavoidable. The realistic choices are:

Keep buying entitlement. Predictable, but unbounded.
Move the cold tail somewhere cheaper that you control, then let Dataverse’s audit-deletion job reclaim the space. The hot months stay in Dataverse where users expect them; the years of historical evidence live in your own storage account, queryable when you need them.

This pattern is for option 2 — specifically, for the one-time bulk export of multi-year history, with the option to keep a slow trickle running afterwards if you want to top up.

A crucial point that often gets lost: this pipeline does not need to run live. It is perfectly reasonable to be deliberately months or years behind real-time. The goal is to get a defensible copy of cold data out — the rows you are about to allow Dataverse to delete — not to mirror the audit feed in real time.

What “good” looks like for an external audit copy

Before showing any code, here’s the rubric I held this design to. If you build your own, hold yours to the same rubric:

Property	Why it matters
Idempotent	Re-running the same time window must not duplicate rows. Network blips happen.
Crash-safe (exactly-once-effective)	If the process dies mid-window, the next run must replay the same window cleanly. The watermark advances only after the data is durable in the destination.
Bounded memory	A backlog of millions of audits cannot be loaded all at once.
Backpressure-aware	Dataverse rate-limits aggressively. Throttle responses must not drop rows.
Observable	Every window logs [entity] mode=BACKLOG/LIVE, lag=Nmin, window=10min, records=N so you can watch it work.
Sink-agnostic	The “where does it land” decision is config, not code. Storage choices change; the orchestrator shouldn’t.
Field-level discretion	Audit details can carry PII. The pattern should let admins narrow which attributes leave the platform.

The pattern

The pipeline is conceptually four stages, repeated per entity, per time window:

Three details in this picture do most of the resiliency work. They are deceptively simple:

Detail 1: Half-open time windows (ge / lt)

The boundary moment (09:10:00.000) belongs to window 2, not window 1. So adjacent windows never overlap and never gap, no matter how many times you replay. This is the same trick Kafka uses for offsets — it’s why you can run the loop with confidence.

Detail 2: The destination document key is the Dataverse auditid GUID

Dataverse already assigns a globally unique GUID to every audit row. That GUID becomes the document id in the sink. So when you upsert the same audit twice, the second write is a no-op overwrite of the first — idempotency for free, no client-side dedupe table to maintain.

Detail 3: The watermark moves after the write, not with it

The naive version of this pipeline does:

The resilient version raises a typed exception when any record fails, and the watermark update is conditional on a clean write:

This single change is the difference between “best effort” and “exactly-once-effective.” It’s also the mistake most often made when people roll their own.

Choosing where it lands

The orchestrator is sink-agnostic — it talks to a single AuditSink interface (get_state, update_state, write_audits) and the destination is a config switch, not a code change. The reference implementation ships with four production-shaped sinks plus a no-op for testing. None of them is the answer; they map to platforms enterprises already operate:

Sink	When to consider it
Azure Cosmos DB (NoSQL API)	Operational lookups — “show me everything user X did to record Y in 2022” in milliseconds. Hierarchical partition keys (/entity + /auditYearMonth) keep partitions small as the archive grows over years. Document TTL doubles as a retention policy if you want one. Serverless mode suits a slow-trickle archive workload.
Azure Data Lake Storage Gen2 (Parquet)	The cheap-cold-storage option. Years of audit history land as partitioned Parquet files (entity=…/year=…/month=…/), readable from Fabric notebooks, Synapse Serverless SQL, Databricks, or any Parquet engine. Costs scale with bytes, not throughput — ideal when the archive is rarely queried but must exist.
OneLake (Parquet)	Same Parquet shape as ADLS, but landed inside a Microsoft Fabric Lakehouse. Immediately queryable from a Fabric SQL endpoint, notebooks, and Power BI without further plumbing. The natural choice if your downstream BI is Fabric.
Snowflake (MERGE INTO)	The natural choice when Snowflake is already the analytics platform of record and adding a separate Microsoft analytics estate just for audit data isn’t on the table. MERGE INTO ... ON audit_id keeps the same idempotency contract as the Cosmos upsert, and the warehouse stays paused between archival batches.
No-op (logs only)	First-day connectivity testing. Confirms the Dataverse side works before you provision any storage.

A reasonable default split many enterprises arrive at:

ADLS Gen2 / OneLake (or Snowflake, if that’s your platform) holds the durable historical archive — cheap, partitioned, queryable when (rarely) needed.
Cosmos DB holds the most recent N months for fast operational lookup if there is a use case for it; otherwise skip it entirely.

Adding a sink for storage you already own (e.g., BigQuery, Redshift, on-prem object storage) is roughly 100 lines of Python and one factory entry.

What real-world numbers will look like

The worked test run above shows what a clean sandbox run looks like. Real numbers in your tenant will vary by orders of magnitude depending on:

How many entities have auditing enabled
The shape of RetrieveAuditDetails calls (more changed attributes per row = more bytes per call)
Dataverse Web API rate limits applicable to your environment
Concurrency you allow (max_concurrent_entities in the config)
Sink throughput (Cosmos serverless RU autoscale, ADLS upload bandwidth, Snowflake warehouse size)

The useful operational signal is not the absolute throughput — it’s that the throughput is stable and the per-window log lines tick predictably. If they don’t, look at lag, sink errors, or 429 responses from Dataverse before scaling up concurrency.

When to reach for it (and when not to)

Reasonable fit:

You have multiple years of accumulated audit history in Dataverse and need to move the cold tail off the platform before pruning to reclaim entitlement.
You want the decoded RetrieveAuditDetails payload (old value → new value, attribute mask, related-record context) landing in the destination ready to query — rather than the packed changedata column Synapse Link delivers, which still needs a parser on the consumer side.
Your analytics platform of record sits outside Azure (most commonly Snowflake) and you don’t want to add Synapse + ADLS + Spark to your stack just to land the audit table.
Synapse Link isn’t an option in your tenant — region pairing, governance review, or the cost floor of running ADLS + a Spark pool 24/7 don’t fit your environment.
You’re comfortable running this as a batch job — once for the historical backfill, then perhaps quarterly or annually to top up — rather than as a live continuous feed. Being deliberately months or years behind real-time is fine and often desirable.
You want field-level control over which attributes leave the platform — useful when audit details contain regulated data.

Not a good fit:

You can run Azure Synapse Link with the Delta Lake profile and you’re happy parsing the packed changedata column on the consumer side, and your destination is ADLS / Synapse / Power BI. That’s the supported, first-class path for the audit table — use it.
You only need current state of business tables (account, contact, opportunity). Use Synapse Link / Fabric Link — they do exactly that and you don’t need this pattern.
You need sub-second freshness in the destination. The pattern’s natural cadence is one window length (10 min in the reference config); for true real-time, use Dataverse webhooks or change-tracking APIs.
You don’t have somewhere to operate a small Python container, function, or scheduled job — even an annual one.
You don’t have an internal owner who can be paged when the schedule fails.

The reference implementation

The code that backs this post lives at https://github.com/SweetsNSavories/DataverseAuditLogSyn under MIT, with no warranty. The unified-deployment folder is the version this post describes — single Python codebase, swap sinks via config.json, runs locally / in a container / as an Azure Function.

If you want the implementation depth this post deliberately leaves out — exact API shapes, watermark math, partial-failure drill, sink-author checklist, hosting variants, observability hooks, and the full list of operational responsibilities a self-hosted export carries — it all lives in one place:

unified-deployment/DESIGN.md

Issues, forks, and pull requests welcome via the repo.

Generated 2026-05-14 from https://github.com/SweetsNSavories/DataverseAuditLogSyn/blob/main/docs/blog/archiving-dataverse-audit-logs.md · MIT licensed reference implementation, no warranty.

Azure OpenAI Architecture: The Decisions That Actually Matter (Part 2)

luciacasucci — Mon, 08 Jun 2026 13:53:24 GMT

Introduction

In Part 1 of this series, we walked through the architectural decisions that shape any Azure OpenAI / Microsoft Foundry Models workload: capacity model, deployment location, governance layer, grounding strategy, and quota engineering. Part 2 moves from decisions to discipline. Once you have made those choices, how do you make sure your design holds up to the Azure Well-Architected Framework (WAF)?

The five WAF pillars — Cost Optimization, Security, Reliability, Performance Efficiency, and Operational Excellence — apply just as strongly to GenAI systems as they do to traditional cloud workloads. In fact, they matter more, because GenAI systems are not static: models are upgraded and deprecated, quotas shift, usage patterns grow unpredictably, and new capacity tiers (such as Priority Processing) are introduced while you are running in production.

This post walks through each pillar in the context of Azure OpenAI in Microsoft Foundry, with best practices, common pitfalls, and the trade-offs Cloud Solution Architects (CSAs) tend to hit in real engagements. Where details are time-sensitive — pricing percentages, SLA windows, model retirement timelines, regional rollout delays — they are flagged with "At the time of writing". Always confirm current behavior against Microsoft Learn before committing to a design.

Who is this series for?

Cloud and Solution Architects
Platform and product owners
Senior developers responsible for operating Azure OpenAI workloads in production

What you’ll learn in Part 2:

How each WAF pillar maps to concrete Azure OpenAI design choices.
Where Priority Processing fits across cost and performance trade-offs (and what its eligibility constraints are).
How to plan for model lifecycle events — upgrades, deprecations, retirements — without firefighting.
Which signals to monitor day-to-day, and how to bake them into a GenAIOps loop.
A WAF Decision Matrix at the end of the article, to use as a reusable checklist.

In Part 3, we will look at the part that makes GenAI architecture genuinely different from a traditional service: the platform itself never stops moving.

We’ve also included a summary decision matrix at the end of this post for quick reference.

1. Cost Optimization: Designing for Sustainable Scale

Cost optimization in GenAI is primarily a capacity strategy problem, not just a token-pricing problem. The first question is whether to use pay-as-you-go capacity, reserved capacity, or one of the newer tiers in between.

Reserved capacity (Provisioned Throughput Units, PTUs)

If your workload is steady or growing predictably, you can significantly reduce costs by reserving capacity up front for 1 or 3 years. At the time of writing, reservations typically yield in the range of 30–50% savings compared to hourly pay-as-you-go rates — but the exact discount depends on term length, region, and the model family, so always confirm against the current Azure pricing page.

Fully utilizing a reserved (provisioned) deployment turns cloud spend into a predictable infrastructure investment, much like allocating VM or database capacity. This requires a mindset shift: treat a provisioned Azure OpenAI deployment as always-on infrastructure sized for peak demand, not as on-demand burst capacity.

Importantly, PTU quota is now model-agnostic within a region. You purchase generic throughput units that can be applied to any supported model in that region, so you do not risk stranded capacity when upgrading (say, from one GPT-4 family version to a newer one) or changing model versions. Your investment follows your architecture, not a specific model endpoint.

Avoid dynamic PTU "auto-scaling"

Unlike VM-based infrastructure, dynamically scaling Azure OpenAI capacity up and down to chase cost savings is not recommended. Additional capacity is not guaranteed to be instantly available when you need to scale up, especially if other tenants are consuming the region's resources. Frequent resizing can also negate the benefits of reservations and introduce performance variability. Unused PTUs are not waste — they are headroom that absorbs burst traffic. In practice, design for the peak load and optimize through reservations rather than trying to constantly dial capacity up and down.

Batch tier

Use Batch deployments for asynchronous, non-user-facing jobs (large-scale document processing, nightly data enrichment, evals, embeddings backfills). At the time of writing, Batch can reduce cost per token by up to around 50% compared to Standard pay-as-you-go calls, in exchange for a 24-hour completion window. It also takes pressure off your real-time deployments.

Priority Processing

For workloads that need prompt responses but do not yet warrant a full dedicated PTU deployment, Azure OpenAI offers Priority Processing. Functionally, it is pay-per-token like Standard, but with SLA-backed lower and more consistent latency on the shared infrastructure.

Activation: set the service_tier attribute on the request to "priority" (other values are "default" and "auto").
Model eligibility: at the time of writing, requires model versions released on or after 2025-12-01.
Deployment eligibility: only available on Global Standard or Data Zone Standard (US) deployments.
Pricing: at the time of writing, roughly 20–40% higher per-token cost than Standard, but still meaningfully cheaper than reserving PTU for a low-volume latency-critical path.

Treat Priority Processing as the natural in-between rung: more predictable than Standard for latency-sensitive production traffic, but without the commitment and capacity-planning effort of PTU.

Putting it together

Segment your workloads by interaction pattern and performance need, then assign the most cost-efficient capacity model to each. A common anti-pattern is over-provisioning expensive real-time capacity for jobs that could run asynchronously. Evaluate whether each use case truly requires sub-second latency, or whether a longer batch window (minutes or hours) is acceptable. Use real-time capacity for customer-facing queries and time-sensitive tasks; use Batch or Priority Processing for everything else, depending on tolerance for latency.

[Diagram 1 — Cost Strategy Layering]

2. Security: Compliance, Isolation, and Data Protection

Security in Azure OpenAI begins with deciding where your inference runs and how data is handled. This is often a compliance-driven decision before it is an architectural one.

Deployment scope

Global deployments — Maximize model availability and capacity by allowing Azure to route inference across regions. Pro: broad elasticity and access to the latest models. Con: data is not confined to a single geography, which may violate strict data residency requirements. Global also adds slight troubleshooting complexity, since requests can be served from various regions.
Data Zone deployments — Constrain inference to a specific zone or political boundary (for example, EU-only or US-only Data Zones). Pro: a compliance-friendly middle ground — data processing stays within a defined region set (for example, entirely within the EU to satisfy GDPR), while retaining more elasticity than a single region. Con: slightly reduced model availability and capacity headroom compared to Global.
Regional deployments — Confine inference to one Azure region. Pro: meets the most stringent data sovereignty requirements and can minimize latency for users in that region. Con: limited to the capacity and models available in one region, with no automatic overflow if the region is saturated. New model versions may also roll out to some regions later than others — at the time of writing we have observed delays of roughly 2–6 months for certain releases in specific regions; check Microsoft Learn for the current rollout schedule.

Choosing among these is a regulatory risk-management decision, not just an infrastructure preference. Engage your compliance and data governance teams early to determine the minimum scope of data movement that satisfies requirements. Many teams initially over-constrain this choice out of caution; it is often better to start with a broader deployment (Global or multi-region Data Zone) where permissible, and tighten the scope later if needed. Conversely, if your organization mandates that all data stay in-country, you might go straight to Regional and invest in architectural mitigation for its limitations (capacity planning, multi-region backup plans).

Baseline protections + defense in depth

Regardless of deployment type, Azure OpenAI provides baseline protections: it does not use your prompts or completions to train the underlying models, and all data is encrypted in transit (TLS 1.2+) and at rest (AES-256). Defense in depth is still essential — implement compensating controls at multiple layers:

Redact sensitive data from prompts (or prevent it from being entered) at the client or gateway layer. Use Azure API Management policies or custom middleware to strip out PII or secrets before requests reach the model.
Apply content filtering to both prompts and responses. Use the built-in content filters and/or Azure AI Content Safety to detect and block sensitive or undesirable input and output.
Use strong authentication and role-based access control. Front your Azure OpenAI endpoint with Microsoft Entra ID; scope tokens with least privilege (for example, the Cognitive Services OpenAI User role or managed identity access) instead of distributing master API keys. If a credential is compromised, the blast radius is limited.

Additional best practices

Managed Identities — use them for any internal communication between your application and Azure OpenAI (or other Azure services like storage and databases) instead of embedding API keys. This eliminates the risk of leaking secrets and simplifies credential rotation.
Private endpoints — enable Azure Private Link to keep traffic between your application and the Azure OpenAI service inside your virtual network and the Azure backbone, off the public internet.
Content Safety tooling — integrate Azure AI Content Safety or custom validation functions to scan prompts and completions for policy violations or confidential data. This extra inspection layer can catch issues the base filter misses, and lets you log or modify disallowed content before it reaches the user.

In short, security for GenAI is not just about encryption or API keys — it is about reducing the blast radius of any potential breach or misuse. Confine inference to approved locations, strip sensitive data before it reaches the model, and strictly limit which identities and networks can call your endpoints.

[Diagram 2 — Data Boundary Visualization]

3. Reliability: Designing for Change, Not Just Stability

Reliability in Azure OpenAI is as much about managing model evolution as it is about traditional uptime. Unlike static services, GenAI models are periodically updated and improved by the provider. New versions are released, older versions are deprecated and eventually retired — so a truly reliable system must plan for these changes just as carefully as it plans for hardware failures.

Model lifecycle

At the time of writing, Generally Available (GA) models are typically supported for at least 12 months after release, followed by a deprecation phase of roughly 6 months before retirement. Always confirm the current support windows on Microsoft Learn before locking in a design — these timelines have shifted in the past and may shift again as new model families ship.

When retirement hits:

Standard deployments still pinned to a retired model (with "No Auto-Upgrade" set) stop responding to requests entirely — the API typically returns HTTP 404 (or a similar error) for that model name.
Provisioned deployments using a retired model return HTTP 410 (Gone) errors until you manually switch them to a supported model.

In short, every model version you deploy has a built-in expiration date. Good reliability planning means never being caught unprepared by a model retirement.

Auto-upgrade modes for Standard deployments

Three modes are available:

Auto-upgrade to the latest version — the deployment moves to the new default model version as soon as Azure makes it available. Always on a supported version, but you have no control over timing. Generally not recommended for mission-critical production workloads, since new versions can have different behavior.
Upgrade only on retirement — the deployment stays on its current version until that version is about to be retired, then automatically switches to the latest. Recommended for most production Standard deployments: stability during the model's supported lifespan, with continuity guaranteed at retirement. You still need to test and adjust to the new version, but at least you do not face an outage if you miss the date.
No auto-upgrade — the deployment stays pinned to a specific version unless you change it manually. Not recommended for production: it puts the entire burden on you to track retirement timelines.

Most teams choose option 2 ("upgrade on retirement") for Standard. It allows controlled change during the model's supported period and provides a safety net at retirement. Proactively evaluate new versions for quality, performance, and cost before the forced swap, but the setting greatly reduces the risk of surprise outages.

Provisioned (PTU) migrations

Provisioned deployments do not support auto-upgrade — you must manage these migrations yourself. Azure sends retirement announcements via Azure Service Health alerts and emails, at the time of writing typically 60 days or more in advance. Have a runbook ready. Two approaches are common:

In-place migration — upgrade the deployment's model version through the portal or CLI. The endpoint stays the same and the model is updated behind it. Fast, no new connection string, but expect a brief disruption during the switch and rollback is not straightforward (you may need to contact support to re-enable the old version, if at all possible).
Side-by-side (blue/green) — create a new deployment with the new model version in parallel. Gradually shift traffic (for example, 10% via APIM routing rules), monitor, and roll back instantly if needed. Maximum control and safety, at the cost of running two deployments in parallel for the migration window.

Before any model migration, verify you have sufficient PTU quota in the region for the new model. More advanced models may require more throughput units for the same workload than smaller predecessors — at the time of writing, plan for the possibility that a new generation needs roughly two times (or more) the PTUs to deliver similar throughput. Request quota increases before you hit the upgrade window, not during it.

Multi-region strategy

Consider a multi-region strategy to improve reliability during model rollouts and deprecations. New model versions do not always appear in all regions simultaneously — at the time of writing, Microsoft often launches a model in one region (frequently East US or West Europe) first.

Maintain a secondary deployment in a "first-wave" region to evaluate new versions early.
Use a traffic manager (Azure Front Door, Traffic Manager) to fail over to a region where the model is still available if your primary region lags behind.
Multi-region active-active designs also protect you against single-region outages.

In essence, reliability for GenAI means designing for change. A highly reliable platform is not one that never changes; it is one that changes gracefully. Model upgrades, deprecations, and capacity adjustments should be routine, well-rehearsed events — not fire drills. Achieving this typically requires automation for detecting and applying updates, redundant deployments or regions for flexibility, and ongoing testing of new models well before your current ones retire.

[Diagram 3 — Model Upgrade Strategy]

4. Performance Efficiency: Predictability Over Raw Speed

Performance in GenAI is multi-dimensional. It is not just about raw throughput or the fastest response on an empty system — it is about consistent, predictable latency at scale. Users care that responses are reliably snappy under load, not just fast in ideal conditions.

Performance profiles by capacity model

Standard (shared infrastructure) — multi-tenant, no guaranteed latency SLA. Performance fluctuates with regional demand; you may see throttling (HTTP 429) at peak. Best-effort: great for development, testing, and non-critical workloads, but not a fit for consistent low latency under spikes.
Priority Processing — also shared infrastructure, but your requests jump the queue ahead of Standard traffic. At the time of writing, this is the only pay-per-token tier with an SLA on latency. Activated by setting the service_tier attribute to "priority" on each request (other values: "default", "auto"). Requires model versions released on or after 2025-12-01 and is only available on Global Standard or Data Zone Standard (US) deployments. Pricing premium is roughly 20–40% over Standard. The natural fit for latency-sensitive workloads at intermediate scale — better than Standard, without committing to PTU.
Provisioned Throughput (dedicated capacity) — reserved capacity with isolation from other tenants. The most consistent performance and the strongest Azure SLA on latency (typically bounded p50 and p99 within your provisioned capacity). If your application has strict response-time requirements or user-facing SLAs and the volume justifies it, PTU is the right answer.

A practical pattern: Standard for early-stage and non-critical scenarios; Priority Processing for latency-sensitive paths that have not yet earned a PTU reservation; PTU for steady, high-volume, latency-critical production traffic.

Model selection and configuration

Model size — smaller models generally respond faster than larger ones. Do not automatically pick the biggest model if a smaller one meets your quality bar.
max_tokens — capping response length caps worst-case latency and cost. A 500-token cap finishes sooner than 2000 tokens, even when users ask open-ended questions.
Sampling parameters — low temperature (more deterministic) and a high top_p can shave a small amount of processing overhead versus highly creative or multi-sample setups. Minor compared to model size and length, but real.
Streaming responses — enable streaming wherever possible. The first tokens arrive immediately while the model is still generating; perceived latency drops dramatically even when total time is unchanged.

Treat performance as an explicit design goal. Choose the right capacity model for the job, tune model settings to avoid unnecessary slowdowns, and do not over-engineer with a larger model than needed. A common mistake is defaulting to the biggest model "just in case". Benchmark — a smaller model with good prompt engineering often delivers a fraction of the latency at a fraction of the cost, with negligible quality loss.

5. Operational Excellence: Running GenAI as a Living System

Operational excellence in GenAI means treating your platform as a continuously evolving product. Models change, user behavior shifts, new features ship. Success requires ongoing monitoring, maintenance, and improvement processes — often called GenAIOps (or MLOps for generative AI).

Proactive monitoring

Set up Azure Service Health alerts for your Azure OpenAI / Foundry resource to be notified about service incidents and, importantly, upcoming model deprecations or retirements. At the time of writing, Microsoft typically gives around 60 days of notice for retirement events — but it is easy to miss those notifications if no one is watching. Early awareness lets you test new models and plan migrations calmly instead of reacting at the last minute.

Continuously track key metrics in Azure Monitor or Application Insights, with alerts on anomalies:

Latency percentiles — monitor p50, p95, and p99. A trend up in tail latency is an early warning of saturation or regression.
Error rates — watch HTTP 429 (throttling) and HTTP 503 (server) error trends. Spikes signal capacity limits or service-side issues.
Capacity utilization — for PTU, watch utilization continuously. Sustained operation near 100% means no headroom for bursts. On Standard, watch token usage against subscription limits and quotas.
Token consumption trends — track growth over time. Helps with cost forecasting and reveals runaway usage (unexpectedly popular features, looping clients).

Useful alerting practices: alert on p99 latency breaching a threshold, on any sustained increase in 429s, or when PTU utilization regularly exceeds around 80%. Early warning lets you scale up, optimize, adjust prompts, or throttle specific users before user experience suffers.

Evaluation and reproducibility

Use the evaluation tooling in Azure AI Foundry to compare outputs from two models side by side on a fixed set of test prompts. Re-evaluate periodically — slowly degrading quality often goes unnoticed without a structured comparison.

Implement Infrastructure-as-Code (IaC) and GitOps practices for your Azure OpenAI and supporting resources (APIM, storage, key vault, monitoring). Bicep, ARM, or Terraform templates checked into source control make environments reproducible across dev/test/prod, simplify recovery, and enable change tracking. If something breaks, you can roll back to a known-good configuration quickly.

In summary, operational excellence for GenAI is about continuous learning and improvement. Embrace an AI DevOps culture: invest in monitoring, train your team on model changes, keep optimizing prompts and configurations, and refine processes after each lesson learned. The effort pays off by preventing fire-drills and keeping the platform robust as it evolves.

Final Perspective and Key Takeaways

Applying the Well-Architected Framework to Azure OpenAI forces a higher level of architectural rigor — exactly what GenAI projects in production need. Each pillar drives concrete decisions.

Key takeaways from Part 2:

Cost Optimization — align capacity to workload patterns. Reserve for steady, predictable load; use Batch for offline jobs; use Priority Processing for latency-sensitive paths that do not yet justify PTU; do not pay for ultra-low latency you do not need.
Security — match deployment scope (Global, Data Zone, Regional) to compliance requirements, then layer controls (network isolation, identity and access, data sanitization, content filtering) to minimize blast radius.
Reliability — anticipate continuous model evolution. Use upgrade-on-retirement for Standard, run parallel deployments for PTU migrations, and design for multi-region failover where it matters. Reliability is about avoiding surprises, not just outages.
Performance Efficiency — choose the right capacity model, right-size models and responses, and use streaming. A smaller model with good prompt engineering often beats a bigger one on user experience.
Operational Excellence — treat the platform as a living product. Monitor, alert, automate, evaluate, and version everything as code. The discipline keeps the platform improving instead of decaying.

The organizations that succeed with Azure OpenAI / Microsoft Foundry Models are those that treat capacity planning, security and compliance, model lifecycle management, and governance as first-class design concerns — not afterthoughts. Generative AI architecture is not about deploying a model and walking away; it is about building a resilient, adaptable platform that gracefully evolves as models change and usage grows.

In Part 3, we bring everything together into a comprehensive reference architecture for an enterprise-grade Azure OpenAI platform — combining scalable capacity strategies, layered security and governance, proactive lifecycle (GenAIOps) practices, and multi-region resiliency into a cohesive blueprint ready for production.

[Diagram 4 — WAF pillars summary]

WAF Decision Matrix : Quick Reference

Use this as a checklist when reviewing or sign-off-ing an Azure OpenAI / Microsoft Foundry Models design. One row per decision; one rule of thumb per row.

Pillar	Decision	Rule of thumb	Watch out for
	Capacity tier mix	Variable load → Standard. Latency-critical → Priority Processing. Offline bulk → Batch. Steady high-volume → PTU.	Single-tier platforms over-pay for elasticity or under-deliver on latency.
Cost	Reservation term (PTU)	1- or 3-year terms for predictable workloads; size for peak, not for average.	Dynamic resizing of PTU to chase savings; capacity not guaranteed at scale-up time.
Cost	Priority Processing eligibility	Requires service_tier="priority", model 2025-12-01+, Global Standard or Data Zone Standard (US).	Assuming it works on every region/model — confirm eligibility before committing the design.
Security	Deployment scope	No residency rule → Global. Multi-region zone OK → Data Zone. Strict residency → Regional.	Over-constraining out of caution; or under-constraining and missing a compliance requirement.
Security	Identity and access	Microsoft Entra ID + Managed Identity + scoped roles. No master keys in apps.	Long-lived API keys distributed across teams.
Security	Network and data	Private Link for in-network traffic; PII redaction at the gateway; content filtering on prompts and responses.	Public endpoints, raw PII in prompts, only relying on the built-in filter.
Reliability	Auto-upgrade strategy	Standard → "upgrade on retirement". PTU → planned blue/green migration with sufficient quota in advance.	Pinning Standard with no auto-upgrade and forgetting; in-place PTU migration with no rollback path.
Reliability	Multi-region	Active-active (or first-wave secondary) for critical paths; traffic manager for failover.	Single-region production with no plan for capacity or model-rollout lag.
Performance	Capacity match	Match capacity tier to latency target: Standard for non-critical; Priority for latency-sensitive; PTU for SLA-bound.	Expecting Standard to deliver consistent low latency under spike load.
Performance	Model and response sizing	Pick the smallest model that meets quality. Cap max_tokens. Stream responses.	Defaulting to the largest model "just in case"; long uncapped responses; no streaming.
Operations	Monitoring and alerting	Track p50/p95/p99, 429/503 rates, PTU utilization, and token trends. Alert on tail latency and sustained throttling.	Average-only dashboards; missed Service Health notifications for model retirements.
Operations	IaC and GitOps	Bicep/ARM/Terraform under source control; reproducible dev/test/prod; pipeline-driven changes.	Click-ops in the portal; environment drift between dev and prod.

Disclaimer

I am a Microsoft employee. The views and opinions expressed in this article are my own and do not necessarily reflect those of Microsoft. This content is informational and educational; it is not an official Microsoft statement, recommendation, or commitment. Service tiers, model availability, pricing, SLAs, and feature eligibility evolve — always validate against the latest Microsoft Learn documentation before making architectural decisions.

References

Azure OpenAI Architecture: The Decisions That Actually Matter (Part 1)

luciacasucci — Mon, 08 Jun 2026 13:53:07 GMT

Generative AI demos often succeed because they hide the hard parts of architecture. They usually run under ideal conditions: low, steady traffic, no sudden bursts, no competing teams, and minimal regulatory scrutiny. In production, however, Azure OpenAI systems face a very different reality – variable loads, service quotas, compliance constraints, evolving model versions, and the need for cost visibility.

The difference between a great demo and a resilient production platform isn’t the model itself – it’s the early architectural decisions. The choices you make from day one determine whether your generative AI solution can handle real-world demand or buckle under pressure.

Who is this series for?

Cloud and Solution Architects
Platform and product owners
Senior developers responsible for operating Azure OpenAI workloads in production

What you’ll learn in Part 1: We’ll walk through five foundational design decisions for Azure OpenAI, explain why they matter, and highlight key trade-offs and pitfalls we’ve seen in real-world deployments:

Capacity Model – Choosing between Standard (PAYGO), Priority Processing, Batch, or Provisioned Throughput (PTU)
Deployment Location – Global vs. Data Zone vs. Regional hosting
Governance Layer – When and why to introduce a GenAI gateway
Grounding Strategy – When to use Retrieval-Augmented Generation (RAG)
Quota Engineering – How to plan for service limits and avoid throttling

In Part 2, we’ll translate these principles into concrete implementations: multi-region topologies, cost allocation strategies, observability and monitoring patterns, and other best practices for reliability, security, and DevOps in Azure OpenAI. Part 3 will connect these decisions to GenAIOps best practices to help ensure your solution is future-proof.

We’ve also included a summary decision matrix at the end of this post for quick reference.

1. Capacity Model: PAYGO, Priority, Batch, or Dedicated Throughput?

At the time of writing, Azure OpenAI (now part of Microsoft Foundry Models) offers four capacity models for hosting models, each with distinct cost, latency, and operational characteristics. Most production solutions combine two or more of these tiers:

Standard (pay-as-you-go, shared): Multi-tenant, elastic capacity. You pay per token, with no upfront commitment and no cost when idle.
Pros: Simple and flexible; ideal for dev/test and for moderate or unpredictable traffic patterns.
Cons: No guaranteed throughput or low-latency SLA – performance may vary with regional load. Under heavy usage, you may see high latency or HTTP 429 “Too Many Requests” errors due to shared capacity limits.
Priority Processing (pay-per-token, SLA-backed low latency): A pay-as-you-go service tier that routes traffic through reserved compute, giving consistent low latency for business-critical, user-facing workloads without buying PTUs. At the time of writing, it is available on Global Standard and Data Zone Standard (US) deployments and requires recent model versions (2025-12-01 or later). It can be enabled per deployment in Microsoft Foundry, or set on individual API calls via the optional service_tier attribute (auto / default / priority) on the chat completions and responses APIs. Always confirm current model and region eligibility in the Microsoft Learn article “Enable priority processing for Microsoft Foundry Models.”
Pros: Predictable, low-latency responses with the simplicity of pay-per-token billing – a strong fit for bursty, business-hour, or latency-sensitive traffic where PTU commitment isn’t justified. Uses the same Standard quota pool, and can be layered on top of PTU for steady-state capacity plus an elastic priority lane for spikes.
Cons: Per-token pricing is higher than Standard PAYGO. Requires eligible deployment types and current model versions, so it is not a drop-in for legacy deployments. Like all PAYGO modes, it is still subject to TPM/RPM limits and the same throttling behaviour if quota is exhausted.
Batch (asynchronous): Offline job processing. You submit requests in bulk (e.g., via a file) and receive results after up to 24 hours.
Pros: Optimized for high throughput at a much lower cost (roughly 50% less per token than real-time requests at the time of writing). Batch jobs use separate “enqueued tokens” quotas, so they won’t interfere with interactive traffic.
Cons: Not suitable for real-time use – no immediate responses or latency guarantees. Requires extra orchestration (staging requests, handling outputs). At the time of writing, Batch does not support embedding models, so vector indexing jobs must use Standard mode – always check the Azure OpenAI documentation for the latest supported model list.
Provisioned Throughput (dedicated PTUs): Reserved, dedicated capacity. You purchase a fixed amount of capacity (Provisioned Throughput Units) and pay for it hourly, whether used or not.
Pros: Guaranteed throughput and consistent low latency, since you’re isolated from other tenants; suitable for high-volume services with strict SLAs.
Cons: Requires careful sizing of PTUs to match your peak demand – under-provision and you’ll still get 429s (now self-inflicted), over-provision and you pay for unused capacity. In addition, output tokens count more heavily against PTU usage (e.g., generative tokens from GPT-4 consume multiple capacity units each), so planning must account for both prompt and completion length.

To right-size your PTU, you’ll need an estimate of the following metrics:

Requests per minute
Average tokens per request
Peak concurrency
Prompt + completion token size

By plugging these into the Azure OpenAI PTU Calculator (linked in the References section), you can get a first estimate of the size of purchase you need to make based on your consumption.

Most production solutions use a hybrid approach: for example, Provisioned capacity for steady, critical real-time traffic, Priority Processing for latency-sensitive bursts that exceed PTU headroom, Standard for overflow or early-stage apps, and Batch for large-scale offline processing. As a rule of thumb: if a user is waiting for a response, use a real-time endpoint (Standard, Priority, or PTU). Use PTU when you need strict latency consistency at high, predictable volume; use Priority Processing when you need SLA-backed low latency without committing to PTUs; use Standard for everything else interactive. If a task can be handled asynchronously, offload it to Batch to reduce cost and keep interactive systems responsive.

Pitfall – Sizing for average load instead of peak burst. One company provisioned capacity only for typical throughput and was overwhelmed when traffic spiked ~3× beyond normal. They maxed out their PTU allocation, triggering a flood of 429 errors. Lesson: model your peak tokens-per-minute (TPM) and requests-per-minute (RPM), not just the average, and add a safety margin to avoid unexpected throttling.

Insight – Separate real-time and background workloads. An initial version of a news analytics bot processed all articles on demand, leading to slow, costly responses. The team later moved heavy processing to Batch jobs (pre-computing article embeddings and summaries), cutting end-user latency by 80% and halving costs.

[Diagram 1 – Capacity Model and Deployment]

2. Deployment Location: Where Does Inference Run?

After choosing the capacity model, decide where your Azure OpenAI instance is hosted. This affects latency, scalability, compliance, and model availability. Azure provides three options:

Global – Your endpoint isn’t tied to a specific region.
Pros: Maximum elasticity and often the best performance stability, since Azure can route traffic to any available regional capacity. You also usually get access to new model releases first on global endpoints.
Cons: Data is processed across multiple regions (may violate strict data residency needs). Troubleshooting can be more complex when calls are served from various locations.
Data Zone – Inference is restricted to a defined geography (e.g., all Azure EU regions).
Pros: Ensures data stays within a specific political boundary for compliance (e.g., GDPR) while retaining some elasticity across multiple regional datacenters in that zone.
Cons: Smaller capacity pool than Global, and possibly a slight delay in getting certain new model versions compared to global rollout. A good balance if you require geographic control without completely sacrificing scalability.
Regional – Inference runs in a single Azure region that you choose.
Pros: Strict data residency and potentially minimal latency if your users are near that region.
Cons: No ability to burst to other regions – you are limited to one datacenter’s capacity. If that region faces high load or an outage, your service is impacted. Some model versions or features may also take longer to become available in a given region than on Global. For current model and region availability, refer to the Azure AI Foundry – Model Deployment Types documentation in the References section.

Pitfall – Over-constraining location without need. Some teams unnecessarily default to a narrow deployment. For instance, a company chose a local Regional deployment out of habit, then discovered the Azure OpenAI model they needed wasn’t available in that region for several months, forcing a last-minute migration to a broader Data Zone. Lesson: unless you have a clear compliance or latency requirement, start with a less restrictive option (Global or multi-region Data Zone) to avoid capacity or availability issues.

Example: One enterprise began with a Global deployment for performance and simplicity, but later had to move to an EU Data Zone to meet GDPR rules, trading some elasticity for compliance. Conversely, a team that started with a single-region setup ran into scaling limits and delayed feature rollouts; they eventually reconfigured to a Data Zone to tap into a larger resource pool.

3. Governance Layer: When a GenAI Gateway Is Needed

As Azure OpenAI usage scales to multiple applications or teams, direct API calls from each app become hard to manage. A central API gateway (such as an Azure API Management instance in front of the OpenAI endpoints) is recommended to enforce enterprise policies and provide a single point of oversight. A gateway enables:

Central Authentication & RBAC: Use Microsoft Entra ID for authentication instead of distributing API keys, and enforce role-based access so each app or team only accesses allowed resources.
Usage Quotas & Throttling: Allocate token or request quotas per application or client. This prevents one service from monopolizing the OpenAI service and can smooth out bursts by applying backpressure (e.g., returning 429s or queueing requests) before Azure OpenAI’s own limits are exceeded.
Intelligent Routing: Direct traffic flexibly – route most requests to a primary model deployment, send a fraction to a new model version (canary), or fail over to a secondary region or the Standard tier if the primary is constrained.
Unified Monitoring & Cost Management: Log all requests in one place. This gives you a clear view of consumption by team or feature, helps with debugging, and supports internal charge-back or cost governance.

A gateway doesn’t make the model faster or more scalable by itself – it’s about control, security, and manageability, not raw performance. That said, for any multi-team or multi-application scenario, a gateway quickly becomes essential to avoid “shadow AI” deployments and chaotic usage patterns.

When to add: Introduce a GenAI gateway once more than one application or team is using the service, or whenever you need to enforce cross-cutting policies. Implementing it early can save a lot of headaches compared to retrofitting it later.

Example: An e-commerce company initially allowed several departments to call the Azure OpenAI API directly. Soon, they had no clear visibility into who was using how many tokens, and costs spiked unexpectedly. They deployed an APIM gateway to require proper authentication, impose per-app quotas, and log usage metrics. The result was rapid identification of the top token-consuming app (preventing it from starving others) and much better cost control.

[Diagram 2 – GenAI Gateway Functionalities and deployment Location sprectrum]

4. Grounding Strategy: When to Use RAG (Retrieval-Augmented Generation)

Many enterprise use cases demand that the model’s answers include specific internal knowledge or citations. Retrieval-Augmented Generation (RAG) is the solution when your AI needs to ground its responses in external data. RAG works by retrieving relevant content from your own data sources and providing it to the model in the prompt:

Document Indexing: First, collect your reference documents (files, knowledge bases, etc.) and break them into chunks, optionally adding metadata (titles, tags). Store these in a vector index or search database after transforming each chunk into an embedding vector.
Relevant Retrieval: For each user query, create an embedding of the question and retrieve the top-matching document chunks from the index via similarity search.
Augmented Prompt: Prepend or append the retrieved text snippets to the model’s prompt (often along with instructions to use them for reference).
LLM Response: The model (e.g., GPT-4) processes the augmented prompt and generates an answer that incorporates the provided reference information (often with source citations if required).

By injecting enterprise data at prompt time, RAG can significantly reduce hallucinations and increase the factual accuracy of outputs. Users get answers that reflect real data you’ve provided, rather than just the model’s training data.

However, RAG adds complexity, latency, and cost. You must maintain additional infrastructure (embedding computation and a vector store or search index). Each query now has extra steps, typically adding 200–500 ms to response time. The vector database and compute for embeddings also incur costs – industry estimates often put a full RAG pipeline at 3–5× the cost of using the base model alone, especially at scale. You’ll need a strategy for keeping your index updated as source data changes, and robust handling for cases where no relevant data is found.

When to use RAG: Use RAG if your model must reliably incorporate proprietary, dynamic, or highly specific information that isn’t part of its training data, or when you need to provide source references for answers. If your scenario is more open-ended or doesn’t require up-to-date factual grounding, you can often skip RAG to keep the system simpler and faster.

Pitfall: Some teams adopt RAG by default, which can slow development and complicate the system unnecessarily. It’s often better to start with a simpler approach and add RAG later if you find the model’s answers need external support.

Examples: One consulting firm added RAG to their internal Q&A bot to leverage proprietary research. Answers became more accurate, but query latency jumped to ~5 seconds due to the retrieval overhead, forcing them to optimize their embeddings pipeline and caching. Conversely, a health company launched a chatbot without RAG and discovered it gave incorrect medical answers because it couldn’t reference the latest policy documents – a failure that a RAG approach could have mitigated.

[Diagram 3 – RAG High Level Flow and Anatomy]

5. Quota Engineering: Avoiding Bottlenecks and Throttling

The Azure OpenAI Service imposes quota limits to protect the system. If you don’t plan for these, they can become points of failure in production. Key limits include:

Tokens per Minute (TPM): Maximum tokens (input + output) your deployment can process per minute (your primary throughput cap).
Requests per Minute (RPM): Maximum number of API calls per minute.
Concurrent Requests: Maximum number of requests processed simultaneously.
Model-specific limits: Certain model types have their own constraints (e.g., the maximum request rate for GPT-4 may be lower than for GPT-3.5 due to higher computational load).

If you exceed these limits, Azure OpenAI will return errors – usually HTTP 429 (Too Many Requests) for quota exhaustion or 503 (Service Unavailable) if the service is stressed. In other words, hitting a quota isn’t a theoretical worry; it will result in rejected requests once you cross the threshold.

Quota Tiers and Deployment Types

Azure OpenAI uses a tiered quota system where limits depend on your subscription’s access level. Specific numeric quotas change frequently, so always verify against the Azure OpenAI Quota Guide (linked in the References section).

Tier 1 (Default): Standard quota allocations suitable for development and moderate production workloads. At the time of writing, GPT-4 deployments in Tier 1 commonly start at quotas in the low tens of thousands of TPM and around a thousand RPM, but exact values vary by region and model – always check the current Azure OpenAI Quota Guide for live numbers.
Tier 2 and Above: Higher quotas available through approval processes, typically for enterprise customers with demonstrated high-volume needs. These tiers can provide significantly more capacity than Tier 1; consult the Quota Guide for current multipliers and approval paths.

Standard (pay-as-you-go) deployments share regional quota pools and are subject to TPM/RPM limits that can vary by region and model. Provisioned Throughput (PTU) deployments operate differently – you purchase dedicated capacity measured in PTUs, and your throughput is determined by your PTU allocation rather than by TPM/RPM limits. PTUs still have implicit rate limits based on the processing capacity of your purchased units.

The Batch API uses a separate quota system with “enqueued tokens” limits, allowing much higher total throughput (often millions of tokens per day) but without real-time guarantees.

Best practices to manage quotas:

Capacity planning: Calculate your peak usage requirements (e.g., max prompt+completion tokens per request × peak requests per minute). Ensure your chosen plan or quota can handle this, or request an increase in advance.
Design for bursts: Traffic often comes in waves. Aim to operate well below your limits so you can absorb sudden surges. As a guideline, keep usage under ~70% of your TPM/RPM limits during normal operation, leaving headroom for peaks. If your usage is spiking above 85% regularly at the 95th percentile, it’s time to scale up capacity or optimize usage.
Graceful degradation: Implement exponential backoff (with jitter) on the client side when retries are necessary. This prevents a stampede of retries (a “retry storm”) that would otherwise compound the load problem. At the platform level, use queues or token-bucket rate limiters (possibly in your APIM gateway) to smooth bursts.
Circuit breakers: Have fallback plans for extreme scenarios. Temporarily disable non-critical features or queue requests when approaching critical limits to prevent a total outage.

Example: A fintech company’s trading chatbot ran fine in testing, but during a market surge their question volume tripled. This breached their tokens-per-minute quota and led to a flood of 429 errors. Worse, their code immediately retried each failed request without delay, intensifying the load and effectively causing a self-inflicted denial-of-service outage. They resolved it by using exponential backoff and partitioning users across multiple deployments.

[Diagram 4 – Quota Engineering]

Final Perspective and Key Takeaways

Ultimately, building a production-grade Azure OpenAI solution is much more about well-structured cloud architecture than about the model itself. An advanced model can underperform in a fragile setup, while even a basic model can excel in a solid architecture.

Key takeaways from Part 1:

Plan for peak loads. Design for the worst-case traffic (and add buffer), not the average. If you need strict performance guarantees, invest in dedicated capacity early.
Avoid unnecessary constraints. Don’t lock into a restricted deployment unless required by compliance or latency – new models and extra capacity reach global deployments first.
Use real-time vs. batch wisely. Real-time endpoints (Standard, Priority, or PTU) should be reserved for interactive, user-facing tasks; move large or non-urgent jobs to Batch for roughly half the cost per token (at the time of writing).
Pick the right real-time tier. Use Priority Processing when you need SLA-backed low latency without committing to PTUs, PTU for high, predictable volumes, and Standard for everything else interactive.
Implement a gateway for scale. If you have multiple applications or teams, use an API Management gateway for authentication, rate limiting, logging, and multi-region routing.
Adopt RAG only if needed. Don’t introduce a retrieval-augmented generation layer unless your application truly demands external data or source citations.
Engineer for limits and failure. Treat rate limits and error handling as fundamental design criteria. Build in monitoring, backoff, and fallback mechanisms so the system degrades gracefully.

In short, succeeding with Azure OpenAI in production means treating it as a full-stack architecture challenge rather than just an API integration. By proactively addressing scalability, deployment, governance, data grounding, and quotas, you can turn a promising demo into a stable, cost-efficient, and compliant AI platform.

In the next part, we’ll explore how to put these principles into practice – including multi-region architectures, cost-sharing strategies for teams, advanced monitoring/logging setups, and other patterns for making Azure OpenAI a robust enterprise service.

Decision Matrix: Quick Reference

Use this matrix as a fast first cut on the five Part 1 decisions. It is not a substitute for a full design review, but it captures the trade-offs most teams need to evaluate up front.

Decision	Choose this when…	Avoid when…	Primary risk
Capacity – Standard (PAYGO)	Dev/test, unpredictable or bursty traffic, MVPs, overflow	You need guaranteed latency or strict SLAs	429 throttling under shared load
Capacity – Priority Processing	Latency-sensitive, business-critical real-time traffic where you don’t want PTU commitment; burst lane on top of PTU	Deployment type / model version isn’t eligible; cost-sensitive, low-priority workloads	Higher per-token cost than Standard; still PAYGO quota-bound
Capacity – Batch	Async jobs, embeddings refresh, large offline summarization	User is waiting; or model is unsupported (e.g., embeddings, at the time of writing)	Up to 24h turnaround; orchestration overhead
Capacity – Provisioned (PTU)	High-volume real-time workloads with strict SLAs and predictable load	Demand is low or highly variable	Over-/under-provisioning costs
Deployment – Global	You want widest model availability and best elasticity	Strict data residency or regulatory constraints	Less control over where data is processed
Deployment – Data Zone	Geographic compliance (e.g., EU/US) with some elasticity	Single-region residency is mandated	Some lag on newest model versions
Deployment – Regional	Strict data residency or co-located low-latency users	You need to scale beyond one region’s capacity	Capacity ceilings and slower model rollout
Governance – GenAI Gateway	Multiple apps/teams, need RBAC, quotas, routing, central logging	Single small app where overhead exceeds benefit	Adds latency and another component to operate
Grounding – RAG	Need proprietary, dynamic, or cited answers	Open-ended creative tasks where freshness isn’t required	Latency, cost, and index freshness drift
Quota – Plan & Tier Up Early	You’re close to TPM/RPM ceilings or expecting growth	Your peak forecast is well below current quota	Last-minute throttling and outages

Tip: If you can only optimize for one decision in Part 1, start with capacity model and quota engineering – they are the two most common sources of production incidents we see in real deployments. Pairing PTUs (or Priority Processing) for steady, latency-sensitive traffic with Standard PAYGO for overflow is a pattern that consistently delivers both reliability and cost control.

Disclaimer

The views expressed in this article are those of the author and do not necessarily reflect the official policy or position of Microsoft. The author is a Microsoft employee.

References

Enabling AI-Driven SAP Development with GitHub Copilot: Plans and Usage-Based Billing Transformation

AnuradhaKarnam — Mon, 08 Jun 2026 13:52:48 GMT

Introduction:

Empower SAP customers with a strategic understanding of GitHub Copilot offerings, including plan options, value realization, account models, and key considerations for enterprise adoption. Through a guided, step-by-step setup in Visual Studio Code, SAP developers will gain hands-on experience with GitHub Copilot and Copilot Chat in real-world development scenarios. In addition, a practical exercise demonstrates how to configure and optimize Copilot within Visual Studio Code enabling SAP teams to effectively adopt AI-assisted development and accelerate application delivery.

Imagine you are an SAP developer working within a fast-paced enterprise environment, tasked with delivering a new feature or extension for a business-critical application, such as SAP S/4HANA or SAP BTP. With limited familiarity with the existing codebase and tight delivery timelines, ensuring quality, performance, and reliability is essential. GitHub Copilot provides a powerful AI-assisted development experience that can help SAP developers accelerate delivery while maintaining high standards. It can interpret complex code, generate documentation, and assist in writing efficient, error-free code enabling teams to meet deadlines and deliver high-quality solutions with confidence.

GitHub Copilot plans for SAP Customers:

Starting June 1, 2026, GitHub is transitioning GitHub Copilot from a request-based billing model to a usage-based billing model. For SAP customers, this shift enables more granular cost alignment with actual AI usage across development teams, supporting improved transparency and budget control within SAP programs.

Organizations and enterprises can leverage this model to better manage consumption across SAP development scenarios such as S/4HANA extensions, BTP applications, and integrations while optimizing costs based on real usage patterns. Individual SAP developers will also benefit from flexible billing aligned to their actual Copilot usage.

GitHub Copilot Capabilities for SAP Development:

GitHub Copilot offers a range of capabilities designed to support SAP developers across the entire development lifecycle from coding and testing to collaboration and deployment.

GitHub Copilot in the CLI: SAP developers can use a chat-like interface directly in the terminal to interact with command-line tools. Copilot provides command suggestions and explanations, helping streamline development workflows for SAP integrations, deployments, and automation tasks.

AI-Generated Pull Request Summaries: Copilot generates intelligent summaries of pull requests, highlighting key changes, impacted files, and areas requiring attention. This enables SAP teams to accelerate code reviews and improve collaboration especially in complex enterprise environments. (Not available in the Free plan.)

GitHub Copilot Extensions
Copilot Extensions allow SAP customers to integrate external tools and systems such as SAP BTP services or other enterprise platforms into the development workflow. These extensions can be customized and shared to support organization-specific SAP scenarios.

Custom Instructions
SAP teams can tailor Copilot responses by providing context about development standards, tools and project requirements ensuring outputs align with SAP best practices and enterprise guidelines.

GitHub Copilot Memory (Public Preview)
Copilot can learn from and retain context about a repository, improving the relevance and quality of suggestions. This is particularly valuable for SAP projects with complex codebases and domain-specific logic.

GitHub Copilot Spaces
Organize SAP project assets such as code, documentation, and specifications into contextual “Spaces.” This ensures Copilot delivers responses grounded in the appropriate SAP business and technical context.

GitHub Copilot in GitHub Desktop
Automatically generate commit messages and descriptions based on SAP code changes, improving consistency, traceability, and developer efficiency.

GitHub Copilot Agents for SAP Development:

GitHub Copilot Chat in Visual Studio Code provides specialized agents designed to support different stages of the SAP development lifecycle ranging from planning to implementation and knowledge exploration.

Agent (Execution Agent)
Use the Agent when you need to implement a specific SAP development task, such as building extensions, integrations, or enhancements for SAP S/4HANA or SAP BTP.
The Agent can autonomously identify relevant files, propose code changes, execute supporting commands, and iteratively refine outputs to complete the task efficiently.

Plan (Planning Agent)
Use the Plan agent when you want to define a structured, step-by-step approach before implementation. This is particularly useful for SAP scenarios requiring careful design such as complex integrations or multi-system workflows. The Plan agent breaks down requirements into clear steps that can be executed by the Agent.

Ask (Knowledge Agent)
Use the Ask agent to explore SAP-related code, development patterns, or general technical concepts without making changes. This is ideal for understanding unfamiliar codebases, learning new SAP development paradigms, or clarifying architecture and best practices.

Implementing GitHub Copilot in SAP Environments:

SAP organizations can follow a structured approach to successfully set up and scale GitHub Copilot across development teams:

Subscribe and Select the Right Plan
Begin by subscribing to GitHub Copilot and selecting the appropriate plan (Business or Enterprise) based on your SAP landscape, governance needs, and scale of adoption.

Establish Governance and Policies
Define organizational policies to control how GitHub Copilot is used across SAP development scenarios. This includes enabling or restricting specific features to align with compliance, security, and SAP development standards.

Configure Secure Networking
Ensure secure connectivity by configuring proxy servers or firewalls to allow required Copilot endpoints. SAP customers may also need to manage SSL certificates to align with enterprise security requirements.

Provision Access to Development Teams
Enable GitHub Copilot for selected SAP developers or teams such as those working on SAP BTP extensions, integrations, or modernization initiatives. A phased rollout can help identify challenges and demonstrate early value before scaling organization wide.

Drive Adoption Across SAP Teams
Establish a structured enablement strategy including training, best practices, and internal champions to promote effective usage of Copilot within SAP development workflows. A self-service licensing model can further accelerate adoption.

Support Developer Authentication
In enterprise environments, ensure SAP developers complete the required authentication steps to access Copilot from their development tools, such as Visual Studio Code.

Enhance the Copilot Experience
SAP organizations can further optimize value by:

Integrating knowledge bases to provide SAP-specific context (Enterprise plan)

Customizing Copilot through tailored AI models aligned to SAP development needs

Leveraging Copilot Extensions to integrate SAP tools and other enterprise systems into the development workflow

Set Up GitHub Copilot in Visual Studio Code:

For SAP developers, setting up GitHub Copilot is a simple process:

Ensure you are using the latest version of Visual Studio Code to support modern SAP development workflows (e.g., SAP BTP and full-stack development).

In the Status Bar, select the GitHub Copilot icon and choose Use AI Features to begin setup.

If your account is already authenticated, select Set up Copilot to complete activation.

Follow the guided steps to authorize Copilot within your development environment.

If no subscription is assigned, SAP developers will be onboarded to the GitHub Copilot Free plan for initial evaluation.

Once setup is complete, GitHub Copilot is ready to assist with SAP development tasks such as code generation, documentation, and optimization.

Customize GitHub Copilot Settings for SAP Development:

SAP developers can tailor GitHub Copilot behavior to align with enterprise development standards and project requirements:

Access Settings by selecting the gear icon in the lower-left corner of Visual Studio Code.

Navigate to GitHub Copilot configurations, organized into key areas:

Code Editing Settings
Configure inline code suggestions, next-edit predictions, and language-specific behaviors to support SAP development languages and frameworks.

Chat Settings
Customize how Copilot Chat behaves, including terminal integration helpful for managing SAP build, deployment, and integration tasks.

Agent Settings
Control advanced capabilities such as agent mode, request limits, and tool approvals to align with SAP governance, security, and compliance requirements.

Selecting Where GitHub Copilot Agents Run in SAP Development:

In addition to selecting the appropriate Copilot agent for a specific task, SAP developers can control where the agent executes using the Agent Target option in the Chat view. This capability determines how the agent interacts with SAP development environments and when results are delivered.

Local Execution
Runs directly within the developer’s environment with full access to the workspace, tools, and codebase. Ideal for SAP developers working on real-time coding tasks, exploring complex SAP codebases, or making controlled updates where step-by-step review is required.

GitHub Copilot CLI (Local Background Execution)
Executes tasks in the background on the developer’s machine. Suitable for SAP scenarios where predefined tasks such as script generation, automation, or batch updates can run while developers continue working on other SAP components.

Cloud Execution
Runs remotely in GitHub and integrates with workflows such as issues and pull requests. This option is well-suited for SAP teams collaborating on large-scale projects, enabling developers to delegate tasks, generate pull requests, and streamline code reviews across distributed teams.

Third-Party Execution
Uses external AI platforms (such as Anthropic or OpenAI) to run agents either locally or in the cloud. This provides flexibility for SAP customers to integrate advanced AI capabilities into their development workflows, depending on enterprise architecture and governance requirements.

Controlling Agent Permissions (for SAP Customers):

You can define how much autonomy your digital agents have by using the Permissions settings within the Chat experience. Adjusting these levels allows you to balance operational efficiency with governance and oversight an important consideration in SAP-driven business processes.

Permission Level	Description
Default Approval	Only low-risk, read-only actions are executed automatically. Most agent-initiated actions require user confirmation, ensuring control over business-critical operations and data changes.
Bypass Approvals	All agent actions are automatically approved without user prompts. This setting streamlines workflows but should be used with appropriate governance controls, especially in sensitive SAP environments.
Autopilot	Enables fully autonomous execution. The agent handles actions end-to-end, including responding to clarification requests, without user intervention. Ideal for well-defined scenarios, but requires strong trust, monitoring, and compliance alignment.

Conclusion:

GitHub Copilot offers SAP organizations a powerful way to modernize development through AI-assisted engineering. It combines intelligent code generation, contextual insights, and advanced features like agents, CLI integration, and memory to accelerate delivery while maintaining enterprise-grade quality and compliance.

It’s usage-based billing model further enhances value by aligning costs with actual adoption, providing transparency, scalability, and control across scenarios such as S/4HANA extensions, SAP BTP applications, and integrations. With a structured approach, covering plan selection, governance, secure configuration, and phased rollout, organizations can integrate Copilot effectively while meeting SAP-specific standards. Ultimately, Copilot enables SAP teams to boost productivity, enhance collaboration, and drive innovation at scale, delivering faster time to market and sustained business value.

Reference links:

Plans for GitHub Copilot - GitHub Docs

Usage-based billing for individuals - GitHub Docs

Usage-based billing for organizations and enterprises - GitHub Docs

Microsoft Mission Critical Blog articles

Managing SharePoint Online Version History at Scale to Recover Storage

Summay

Why version history planning matters

What Microsoft recommends admins consider

Automatic vs. manual version limits

What the tool does

Why I built it as an admin workflow

Site discovery options

Key operations:

Get current version policies

Set version policies across sites

Check version policy status

Generate version history reports

Run What-If analysis

Create batchdeletejobs

Monitor batchdeletejob status

Tenant-level defaults vs. existing content

Microsoft's guidance describes multiple approaches:

A practical rollout approach

Where this helps most

Permissions and prerequisites

The repository README includes setup details and permission guidance:

Safety notes

Why this belongs in your governance toolkit

Final thoughts

More Information

AI Just Joined Your Code Reviews: GitHub Copilot Comes to Azure Repos

Overview

Prerequisites

Enable Copilot code review

Configure reviews

Automatic reviews & instructions

Request and use a review

Scope and limits

Billing

Recommended practices

Troubleshooting

Visual walkthrough

Choosing a review effort

The honest scorecard

The bottom line

Sources and further reading

The new sharing experience is coming to SharePoint Online: What admins need to know

Meet the hero link

What changes in the sharing interface

Sharing UI

Sharing settings

Link scope

Expiration links

Stop sharing

Manage Access

Advanced settings

Key takeaways for end users

First-run experience

Customer communications

More information

Known issues and current limitations

FAQ

What’s changing with sharing?

What problem does it solve?

How does the new sharing link change my governance model?

Will any existing links or permissions break during rollout?

What if a link leaks?

How does this interact with information barriers?

Why does the new sharing link matter for Copilot?

Does the new sharing link reduce oversharing risk for Copilot?

Final takeaway

Your on-call rotation has a new member: 10 production incidents, end to end, with Azure SRE Agent

Table of Contents

TL;DR

1. Why the "investigation" half is the real prize

2. What Azure SRE Agent actually does

The three primary use cases

The five extension primitives

Integrations you can assume

Skills vs. custom agents vs. knowledge files

3. Anatomy of an agent-run incident

4. Build the demo lab

4.1 Resource group and agent