Managing SharePoint Online Version History at Scale to Recover Storage
Summay SharePoint Online version history is one of those features that users love, but admins eventually have to govern. Versioning protects users from accidental edits, bad changes, and overwritten content. It gives people a way to restore previous versions of documents without opening a ticket. That is a good thing. But at tenant scale, version history can also consume a surprising amount of storage. Highly active libraries, large Office files, frequent edits, migrations, sync activity, and years of retained versions can quietly add up. Eventually, admins need to answer a few practical questions: How many file versions do we have? How much storage are those versions consuming? Which sites would benefit most from cleanup? What policy should we apply going forward? How much space could we recover before making changes? How do we clean up versions safely across many sites? That is why I built this tool: SPOVersions-ApplicationTools The main script is: Apply-SPOVersions-Tool.ps1 Why version history planning matters Microsoft's official guidance for SharePoint version storage makes an important point: version limits should support your organization's recovery objectives while also controlling storage consumption. Microsoft documents this here: Plan version storage for document libraries in SharePoint The key idea is balance. If you keep too many versions for too long, storage consumption can grow quickly. If you trim too aggressively, users may lose the ability to recover from accidental or unwanted changes. A good version history strategy needs to consider both: Goal Why it matters Recovery Users need enough version history to restore useful prior versions. Storage control Admins need to prevent version history from consuming excessive quota. Consistency Tenant defaults and site-level policies should be understandable and repeatable. Exceptions Some business areas may need different version retention behavior. Safe cleanup Existing versions should be analyzed before deletion jobs are created. The goal is not simply to delete versions. The goal is to understand version storage, set the right policy, and clean up intentionally. What Microsoft recommends admins consider Microsoft's documentation calls out three major planning questions: Determine the default version history limits for new document libraries. Identify exceptions needed from organization default limits. Decide whether existing content should be updated to align with the organization limits. That third point is where many admins need tooling. It is one thing to configure a tenant default for new libraries. It is another thing to understand what already exists across hundreds or thousands of sites. Existing content is often where the storage opportunity is. Automatic vs. manual version limits SharePoint provides different ways to manage version history. Mode What it does Best fit Automatic SharePoint uses intelligent trimming based on version age. Recent versions are preserved more densely, while older versions are thinned over time. Recommended default for many organizations that want Microsoft-managed optimization. Manual with count and expiration Admins specify how many major versions to keep and how long versions should remain. Useful when you want more direct control and predictable cleanup. Manual with count only Admins keep a configured number of versions without age-based expiration. Useful when a fixed restore count matters, but it can consume more storage. Microsoft's guidance highlights that automatic trimming is designed to preserve high-value versions while optimizing storage. Manual policies can be useful, but they require careful planning because low limits or short expiration windows can affect restore scenarios. That is why I wanted this tool to include reporting and What-If analysis before cleanup. What the tool does Apply-SPOVersions-Tool.ps1 is an interactive PowerShell tool built with PnP PowerShell to help admins manage SharePoint Online version policies across multiple site collections. At a high level, the tool helps admins: Audit current version policies across sites. Apply automatic or manual version limits. Review version policy status. Generate version history reports. Run What-If analysis to estimate storage recovery. Create batch delete jobs for cleanup. Monitor cleanup job status. Set tenant-level defaults for new sites. This makes it useful for both planning and implementation. Why I built it as an admin workflow Version cleanup should not start with deletion. The safer workflow is: Discover current settings. Generate version history reports. Estimate storage recovery. Decide the right policy. Apply policy to selected sites. Queue cleanup jobs. Monitor job status. Repeat as part of governance. That is how the tool is structured. It is not just a cleanup script. It is a planning and execution tool. Site discovery options The tool can work from a defined list of site URLs or discover sites automatically. For larger tenants, I usually prefer a site list because it gives admins better control over rollout scope. You can start with a pilot group, validate results, then expand. For smaller tenants or discovery scenarios, auto-discovery can be useful. The tool supports: SharePoint sites OneDrive for Business sites Exclusion of common system sites Confirmation before processing discovered sites That confirmation step matters. When a script can operate across many sites, admins should have a checkpoint before changes are made. Key operations: Get current version policies Before changing anything, admins can retrieve current version policy settings for all selected sites. This helps answer: Which sites already use automatic trimming? Which sites use manual limits? Which sites have inconsistent settings? Which sites may need exceptions? This is useful for building a baseline. Set version policies across sites After planning, admins can apply version policies across site collections. The tool supports: Automatic version trimming Manual major version limits Optional expiration after a specified number of days Automatic trimming is useful when you want SharePoint to optimize storage intelligently. Manual limits are useful when the organization has a defined policy, such as keeping a specific number of major versions or expiring versions after a specific period. Check version policy status Policy changes may take time to apply. The tool can check propagation status, so admins can verify whether pending changes are completed successfully. This is important during staged rollouts because you want to confirm policy state before moving on to cleanup. Generate version history reports This is one of the most important parts of the workflow. The tool can submit version history report jobs for each site. Those reports are written to a dedicated document library created by the tool: Admin_SiteCollection_VersionReport_DONOTDELETE Using a dedicated library avoids some of the issues that can happen when report files are written into normal user-facing document libraries. The reports provide the raw data needed to understand version storage usage. Run What-If analysis This is the feature I think most admins should start with. The What-If analysis downloads completed version history reports and simulates a version policy without making changes. That means admins can estimate: Total versions analyzed Versions that would be deleted Total version storage Storage that could be recovered Percent of version storage that could be freed Per-site savings This gives admins data before they commit to a cleanup policy. For example, a What-If analysis might show that one site can recover a large amount of storage while another site has very little benefit. That helps prioritize action. Create batchdeletejobs Once the admin understands the impact and chooses a policy, the tool can create batch delete jobs to remove excess versions. This is the destructive part of the workflow, so it should come after reporting and What-If analysis. Admins can use automatic or manual cleanup criteria depending on the desired policy. Important reminder: deleted versions are not recovered from the Recycle Bin. Treat batch delete jobs as permanent cleanup operations. Monitor batchdeletejob status After cleanup jobs are created, admins need to know whether they completed and how much storage was released. The tool can check batch delete job status across sites, including completion state and storage released where available. This closes the loop from planning to execution. Tenant-level defaults vs. existing content One important distinction in SharePoint version management is the difference between tenant-level defaults and existing content. Tenant-level settings affect new sites or new libraries going forward. Existing sites and existing file versions may need additional steps. Microsoft's guidance describes multiple approaches: Scenario What it means Apply defaults to new libraries only Least disruptive; existing settings and versions remain unchanged. Update existing library settings without trimming versions Future versions follow the new policy, but existing versions are not deleted. Update settings and trim existing versions Aligns current and future behavior but requires careful planning and cleanup jobs. This tool helps with the third scenario, where admins need to understand and manage existing version storage across sites. A practical rollout approach For most tenants, I would not recommend starting with the entire environment. A safer rollout looks like this: Phase Action Phase 1: Baseline Get current version policies across a small set of sites. Phase 2: Report Generate version history reports for pilot sites. Phase 3: What-If Estimate storage recovery under automatic or manual policies. Phase 4: Decide Choose default policy and identify exceptions. Phase 5: Apply Apply policy to selected pilot sites. Phase 6: Cleanup Queue batch delete jobs only after review. Phase 7: Monitor Check job status and storage released. Phase 8: Expand Repeat across additional sites or workloads. This creates a defensible admin process instead of a one-time cleanup event. Where this helps most This tool is especially useful in tenants with: Large document libraries Heavy collaboration sites Long-running project sites Migration history Large Office files with frequent edits Storage pressure Inconsistent site-level version policies Need for tenant-wide governance reporting It can also help organizations that want to move toward automatic trimming but need to understand the impact first. Permissions and prerequisites The tool uses PnP PowerShell and interactive authentication. Typical prerequisites include: PowerShell 5.1 or later PnP.PowerShell 3.1.0 or later SharePoint Administrator role Entra ID app registration Delegated SharePoint permissions required for version policy and report operations Important: Ensure that the account running this tool has site collection admin rights on all sites If you need to add your Admin account to all OneDrive and SPO sites, check out this scripts: Add-GroupToSPOSites.ps1 Add-GroupToOneDriveSites.ps1 The repository README includes setup details and permission guidance: Apply-SPOVersions-Tool README As always, review permissions carefully before using any tenant-wide admin automation. Safety notes Version cleanup can permanently remove version history. Before creating cleanup jobs: Run reports first. Run What-If analysis. Start with pilot sites. Review business requirements. Identify exceptions. Communicate expected behavior to site owners. Keep exported results for audit and planning. Avoid opening or modifying report CSV files while report jobs are still running. The tool is designed to support safe planning, but admins still need to make policy decisions carefully. Why this belongs in your governance toolkit SharePoint storage management is not just about quota. Version history touches: User recovery expectations Records and retention requirements Collaboration patterns Site lifecycle Migration cleanup Storage forecasting Governance consistency If version history is unmanaged, storage growth can feel mysterious. Once you report on it, it becomes something you can plan and control. That is the point of this tool: help admins move from guessing to measuring, then from measuring to controlled action. Final thoughts SharePoint version history is valuable, but it needs governance at scale. Microsoft provides flexible version storage options, including automatic trimming and manual limits. The challenge for admins is understanding what exists today and how much storage could be recovered before applying changes. Apply-SPOVersions-Tool.ps1 helps with that process by combining policy discovery, report generation, What-If analysis, policy application, cleanup jobs, and job monitoring into a single admin workflow. More Information You can find the tool here: SPOVersions-ApplicationTools Microsoft's official planning guidance is here: Plan version storage for document libraries in SharePoint Suggested Reading for More Information: SharePoint Automatic Version History Cleanup (Intelligent Versioning)
Your on-call rotation has a new member: 10 production incidents, end to end, with Azure SRE Agent
What this post is. A hands-on, reproducible walkthrough of ten real production failure modes — App Service, AKS, Azure SQL, Cosmos DB, VMs, VM Scale Sets, Application Gateway, and Service Bus — each one driven end to end by Azure SRE Agent: detection, hypothesis-driven investigation, ITSM ticketing, bounded remediation, recovery validation, and follow-up records. What this post is not. A claim that you install SRE Agent and all of this happens on day one. Every workflow below needs telemetry, scoped RBAC, a response plan, an approved action surface, and an ITSM integration. I'll be explicit about which parts are documented product behavior and which parts you have to wire yourself — because that distinction is the difference between a demo that works on stage and one that works at 3 AM. TL;DR The pattern Alert → agent investigates read-only → agent opens/updates the ticket with evidence → agent proposes a bounded action → human approves → agent executes → agent validates recovery → agent files the follow-up record The unlock Not "AI fixes prod." The unlock is that investigation — the 20 minutes of tab-switching between Azure Monitor, App Insights, deployment history, and Activity Log — is fully automated and consistent, and the fix arrives pre-justified with an evidence chain The guardrail Read-only automatic. Ticketing automatic. Production writes in Review mode. Guest-OS work through fixed-purpose runbooks, never a shell prompt The reality check SRE Agent hard-blocks delete / remove and all az keyvault commands, respects Azure management locks, and only one incident platform can be active at a time. Several patterns in this post need a custom tool or MCP server to complete the loop Time to first value One resource group, one alert rule, one response plan. You can reproduce use case #1 in an afternoon 1. Why the "investigation" half is the real prize Every conversation about AI in operations goes straight to remediation. "Will it restart my app?" That's the least interesting question, and it's the one with the most downside risk. Think about what actually consumes the minutes during a Sev1. The alert fires. Someone acknowledges. Then: Open Azure Monitor, confirm the metric is real and not a probe artifact Open Application Insights, find the dominant exception Open the deployment pipeline, find what shipped and when Open Activity Log, check whether someone changed configuration Open Resource Health, rule out a platform incident Open the other environment, confirm the previous version is healthy Assemble all of that into a sentence a human can act on That's twenty minutes of context assembly performed by a tired human, differently every time, with quality that depends entirely on who happens to be on call. It is the single most automatable part of incident response and the part nobody automates, because scripts can't reason about which of six hypotheses fits the evidence. This is exactly what root cause analysis in SRE Agent is designed for. The agent doesn't grep logs — it forms hypotheses and invalidates them: HYPOTHESIS 1: Recent deployment broke something ├─ Checked: Last deployment was 3 days ago ├─ Evidence: Error rate stable until 30 minutes ago └─ Result: INVALIDATED HYPOTHESIS 2: Database overloaded ├─ Checked: Azure SQL metrics (CPU, DTU, connections) ├─ Evidence: DTU at 98%, query duration 4x normal ├─ Traced: SELECT * FROM orders WHERE... taking 8.2s └─ Result: VALIDATED ROOT CAUSE: Orders table missing index on customer_id column. Query plan shows full table scan on 2.1M rows. RECOMMENDED ACTION: Add index on orders.customer_id Similar fix applied in INC-2341 (3 weeks ago) That last line — recalling a similar incident from three weeks ago — is the compounding part. Every thread produces a session insight: symptoms observed, steps that worked, root cause, and pitfalls to avoid. Thirty minutes after a thread goes quiet, the agent indexes those learnings. Next time the same resource misbehaves, that history surfaces first. So the framing for the rest of this post: remediation is the punchline, but investigation is the product. Every one of the ten use cases below has a large read-only phase you can turn on tomorrow with zero write permissions, and a small write phase you should gate behind approval for a long time. 2. What Azure SRE Agent actually does Before the use cases, here is the honest capability map, drawn from the product documentation rather than from a keynote. The three primary use cases Use case What it means Automate incidents Alert fires → agent queries monitoring tools, correlates signals across systems, identifies probable root cause, proposes mitigations Automate scheduled workflows Proactive health checks, compliance sweeps, and routine tasks on a schedule, with results routed to your incident platform or notification channel Investigate and advise Natural-language questions — "what changed in the last hour?" — answered with grounded citations The five extension primitives Everything you customize sits in one of five buckets: Primitive What it is When you reach for it Skills Procedural guidance ( SKILL.md ) plus optional attached tools; auto-loaded when relevant Team troubleshooting runbooks that should also execute Subagents / custom agents Purpose-built specialists invoked via /agent or routed to by a response plan A DatabaseExpert that owns every SQL incident Python tools Custom logic, transformations, API calls Anything that needs code, e.g. writing to the ServiceNow Table API MCP servers 40+ managed connectors (Datadog, New Relic, Splunk, Elastic, Dynatrace…) plus any custom MCP tool Bringing your non-Azure telemetry and your ITSM write surface into the loop Agent hooks Event-triggered automations at Stop and PostToolUse Policy enforcement, audit emission, blocking risky commands Six generic subagents ship built in — Explore, Plan, CodeReview, Bash, Verification, GeneralPurpose — and the agent can parallelize investigation, planning, review, shell, and verification work across them. A permission gate sits in front of all five primitives and evaluates every proposed tool call before it runs. Integrations you can assume Category Supported Monitoring Azure Monitor (metrics, logs, alerts, workbooks), Application Insights, Log Analytics, Grafana Incident platforms Azure Monitor Alerts, PagerDuty, ServiceNow — only one active at a time Source control / CI GitHub (repos, issues), Azure DevOps (repos, work items) Data Azure Data Explorer (Kusto), MCP servers Comms Slack, Microsoft Teams ⚠️ Design constraint worth internalizing early. Only one incident platform can be active at a time, and switching disconnects the current one. If your org runs PagerDuty for paging and ServiceNow for records of truth, you must pick which one the agent is bound to and reach the other through a connector or custom tool. Every use case below assumes ServiceNow is the bound platform. Skills vs. custom agents vs. knowledge files The three are constantly confused. This table settles it: Skills Custom agents Knowledge files Access Automatic when relevant Explicit ( /agent ) or routed by response plan Automatic search Tools Can attach Has its own None Context Uses thread context Shares thread context (no clean slate) Reference only Best for Team procedures with execution Domain specialists Runbooks, architecture docs Practical limits: a maximum of five concurrent active skills (oldest auto-unloads), knowledge base uploads up to 16 MB per file, and custom agent knowledge bases up to 1,000 files. 3. Anatomy of an agent-run incident Every use case in section 6 is an instance of this one shape. Learn it once. [[INCIDENT_FLOW_IMAGE]] Five properties make this shape safe, and they're worth stating as rules: The read phase has no blast radius. Turn it on everywhere, immediately, with Reader . One action per incident. Not "roll back and scale and restart." One bounded, reversible step, then re-measure. The proposed action is always the smallest reversible one. Swap a slot, don't redeploy. Bump one service tier, don't resize the cluster. Validation is a first-class phase, not a vibe. Define the metric, the threshold, and the duration before you approve. Technical recovery ≠ business recovery. Use case #10 makes this painfully clear. 4. Build the demo lab Everything below runs in a single throwaway resource group. Nothing here should touch a subscription you care about. 🧪 Lab hygiene. Create it, demo it, delete it. az group delete -n rg-sre-agent-demo --yes --no-wait when you're done. Several of these use cases deliberately break things. 4.1 Resource group and agent LOC=eastus2 RG=rg-sre-agent-demo az group create -n $RG -l $LOC Create the SRE Agent from the Azure portal and point it at $RG . Three things are created for you automatically: an Application Insights instance (this is where your audit trail lands), a Log Analytics workspace, a user-assigned managed identity (UAMI) — the identity every action runs as. 4.2 Choose a permission level deliberately At creation you pick one of two levels: Level Grants Use it when Reader Core monitoring roles + resource-type reader roles. Prompts for temporary elevation via on-behalf-of when it needs to act Start here. Production. Weeks 1–4 of any pilot Privileged Core monitoring roles + resource-type contributor roles Non-production, or after a proven pilot Regardless of level, these are always assigned: Role Scope Why Reader Resource group See resources and properties Log Analytics Reader Resource group Query logs and workspaces Monitoring Reader Resource group Read metrics Monitoring Contributor Subscription Acknowledge and close Azure Monitor alerts Grant additional access explicitly and narrowly: SUB=$(az account show --query id -o tsv) AGENT_MI=<agent-managed-identity-principal-id> # Read everywhere you want visibility az role assignment create \ --assignee $AGENT_MI \ --role "Reader" \ --scope "/subscriptions/$SUB" # Write ONLY where you intend the agent to act az role assignment create \ --assignee $AGENT_MI \ --role "Website Contributor" \ --scope "/subscriptions/$SUB/resourceGroups/$RG/providers/Microsoft.Web/sites/app-checkout-demo"</agent-managed-identity-principal-id> 📌 A sharp edge. You can't remove individual permissions from an agent — only entire resource groups. Removing a resource group from the agent's scope revokes all access to it. Plan your resource group boundaries as your blast-radius boundaries, because that's exactly what they are. 4.3 Connect ServiceNow Use a dedicated, least-privileged integration user — not a shared admin account. Method When What you need Basic auth Quick setup, PDI, testing Username + password, itil or admin role OAuth 2.0 Production ServiceNow OAuth app (client ID + secret); register redirect https://logic-apis-{region}.consent.azure-apim.net/redirect Scope the connection so you don't drown: Assignment group — essential on a shared enterprise instance Priority — Critical through Planning Category Scanner defaults worth knowing: Setting Value Scan interval 1 minute Incidents per page 20 Max incidents per cycle 220 (11 pages) Initial lookback 30 days Setup performs a real connectivity check by fetching an actual incident, so credential and endpoint mistakes surface immediately instead of six hours later when nothing syncs. 🚨 Delete the quickstart plan. Connecting an incident platform auto-creates a quickstart_handler response plan that runs in fully autonomous mode across all impacted services. If you then build your own plans, incidents get routed twice or to the wrong agent. Go to Builder → Incident response plans → Table view and delete it before you do anything else. 4.4 Connect deployment correlation Half the use cases below hinge on "what shipped three minutes before the spike." That correlation requires a source control connector: GitHub — repositories and issues Azure DevOps — repos and work items Connect one. Without it, the agent can still read Azure Activity Log and deployment history, but it can't reach commits, PRs, or work items. 4.5 Create the demo resources # 1 · App Service with a staging slot (use cases 1) az appservice plan create -g $RG -n plan-demo --sku P1V3 --is-linux az webapp create -g $RG -p plan-demo -n app-checkout-demo --runtime "DOTNETCORE:8.0" az webapp deployment slot create -g $RG -n app-checkout-demo --slot previous # 2 · AKS (use case 2) az aks create -g $RG -n aks-demo --node-count 3 --generate-ssh-keys --enable-addons monitoring # 3 · Azure SQL (use case 3) az sql server create -g $RG -n sqlsrv-sre-demo -u sqladmin -p "<use-a-generated-password>" az sql db create -g $RG -s sqlsrv-sre-demo -n db-customer --service-objective S1 # 4 · Cosmos DB (use case 4) az cosmosdb create -g $RG -n cosmos-sre-demo az cosmosdb sql database create -g $RG -a cosmos-sre-demo -n catalog az cosmosdb sql container create -g $RG -a cosmos-sre-demo -d catalog \ -n products --partition-key-path /category --throughput 400 # 5–7 · VMs (use cases 5, 6, 7) az vm create -g $RG -n vm-payments-linux --image Ubuntu2204 --size Standard_B2s --generate-ssh-keys az vm create -g $RG -n vm-claims-win --image Win2022Datacenter --size Standard_B2ms \ --admin-username azureadmin --admin-password "<use-a-generated-password>" # 8 · VM Scale Set (use case 8) az vmss create -g $RG -n vmss-api-demo --image Ubuntu2204 --instance-count 12 \ --upgrade-policy-mode automatic --generate-ssh-keys # 10 · Service Bus (use case 10) az servicebus namespace create -g $RG -n sb-sre-demo --sku Standard az servicebus queue create -g $RG --namespace-name sb-sre-demo -n order-events \ --enable-dead-lettering-on-message-expiration true</use-a-generated-password></use-a-generated-password> Install the Azure Monitor Agent on the VMs and associate a data collection rule — without guest telemetry, use cases 5, 6, and 7 have nothing to detect. 4.6 One custom agent per domain Don't build a single mega-agent. Build specialists and let response plans route to them: Custom agent Owns Attached tools DeploymentAnalyzer App Service, AKS, anything release-correlated RunAzCliReadCommands , GitHub connector, Kusto DatabaseExpert Azure SQL, Cosmos DB RunAzCliReadCommands , Kusto, read-only SQL diagnostics tool GuestOSResponder VM / VMSS / IIS RunAzCliReadCommands , fixed-purpose runbook tools only NetworkPathExpert Application Gateway, NSG, DNS, TLS RunAzCliReadCommands IntegrationExpert Service Bus, Event Hubs RunAzCliReadCommands , Kusto A custom agent definition is small: name: database_expert system_prompt: | You are a database specialist for this estate. Analyze query performance, diagnose connection and saturation issues, and recommend the smallest reversible mitigation. Never propose schema changes, index changes, plan forcing, or session termination — those are DBA-owned and require a change record. handoff_description: Handles Azure SQL and Cosmos DB troubleshooting tools: - execute_kusto_query - RunAzCliReadCommands allowed_skills: - azure-sql-saturation-runbook - cosmos-throughput-runbook Note what that system_prompt is really doing: it is narrowing the action space. Half of production safety with an agent is telling it, in plain English, which categories of fix are off the table. 4.7 Response plans Create one plan per domain, all starting in Review: Plan Filter Custom agent Mode appsvc-p1 Priority 1 + 2, service checkout DeploymentAnalyzer Review aks-p1 Priority 1, service orders-api DeploymentAnalyzer Review db-critical Priority 1 + 2, service customer-api DatabaseExpert Review vm-guest Priority 1 + 2, title contains disk , memory , CPU GuestOSResponder Review network-p1 Priority 1, title contains 502 , backend NetworkPathExpert Review integration-p1 Priority 1, service fulfillment IntegrationExpert Review Filters available: severity/priority (multiselect), impacted service, incident type, and title contains. Plans can be turned off without deleting them, which is exactly what you want during maintenance windows. 5. Set your guardrails before your first incident If you read only one section of this post, read this one. The guardrails are not paperwork; they are the reason this is deployable. 5.1 Run modes Mode Behavior Default Review Agent proposes; you approve or deny Agent-level default Autonomous Agent executes immediately and reports Per-plan and per-task default Two subtleties that bite people: Run modes are set per response plan and per scheduled task, not globally. The agent-level setting is only a fallback. And per-plan the default is Autonomous — so if you don't set it, you get autonomy. Review mode shows Approve/Deny only for Azure infrastructure operations (Azure CLI, ARM writes). Sending an email, posting to Teams, or querying an external source proceeds based on the agent's reasoning. To gate those, you need hooks or tool access policies. Only users holding the SRE Agent Administrator role can approve. Standard users cannot, and personal Microsoft accounts can't authorize on-behalf-of at all — it requires a work or school (Entra ID) account. 5.2 What the product blocks for you These are enforced at the command level, independent of your RBAC: Guardrail Behavior Delete operations The agent never runs delete or remove commands. It returns an error pointing you at the portal Key Vault All az keyvault commands are blocked, to prevent credential exposure Management locks Resources with ReadOnly locks can't be modified, regardless of permissions or run mode Subscription validation Subscription IDs are validated as well-formed GUIDs before execution 💡 Use the delete block architecturally. Put a ReadOnly management lock on anything that must never change during an incident — your Key Vaults, your production databases, your golden images. That lock is respected before every modification, which gives you a control that survives RBAC drift and misconfigured response plans. 5.3 Hooks: the guardrail you write yourself Two events are supported — Stop (agent about to return a final response) and PostToolUse (a tool finished). Hooks run as an LLM prompt or a sandboxed command script. Here is the single most useful hook for this entire post — a deterministic policy gate on shell execution: hooks: PostToolUse: - type: command matcher: "Bash|ExecuteShellCommand" timeout: 30 failMode: block script: | #!/usr/bin/env python3 import sys, json, re context = json.load(sys.stdin) command = context.get('tool_input', {}).get('command', '') dangerous = [ r'\brm\s+-rf\b', r'\bsudo\b', r'\bchmod\s+777\b', r'\bmkfs\b', r'\bdd\s+if=', r'\btruncate\b', r'\bDROP\s+TABLE\b', ] for pattern in dangerous: if re.search(pattern, command, re.IGNORECASE): print(json.dumps({"decision": "block", "reason": f"Blocked by policy: {pattern}"})) sys.exit(0) print(json.dumps({"decision": "allow"})) And a Stop hook that refuses to let the agent declare victory without evidence: hooks: Stop: - type: prompt model: ReasoningFast timeout: 30 prompt: | Review the agent's final response. $ARGUMENTS It is only acceptable if it contains ALL of: 1. The specific resource acted upon (full name or resource ID) 2. Metric values BEFORE and AFTER the action 3. The observation window over which recovery was confirmed 4. Who approved the action and at what UTC timestamp Respond with: {"ok": true} {"ok": false, "reason": "<what is="" missing="">"}</what> Hook mechanics you'll want on hand: Setting Default Range / notes timeout 30s 1–300s failMode allow allow or block maxRejections 3 1–25; prompt-type Stop hooks only matcher — Regex, anchored ^(pattern)$ , case-sensitive; * matches all Script size — 64 KB max Shebangs — #!/bin/bash , #!/usr/bin/env python3 ⚠️ The footgun: for Stop hooks, a rejection without a reason field is treated as approval. Always populate reason . Agent-level hooks and custom-agent-level hooks both run when both match; agent-level fires first. 5.4 The recommended production policy This is the policy I'd put in front of a change board: Allow read-only investigation automatically, everywhere. Allow automatic incident creation and work-note updates — after sanitization. Use Review mode for all production remediation. Require human approval for every production write. Use fixed-purpose runbooks instead of unrestricted VM shell access. Require separate approval for destructive or data-affecting actions. Initially require human confirmation before resolving an incident. 6. The ten use cases Each use case follows the same seven-part structure so you can skim to the one you're firefighting: What happened → How Azure Monitor detected it → How the agent found the cause → Agent action table → Recovery note → Try it yourself → Guardrails In the action tables, Type is one of: Type Meaning Read No blast radius. Safe to run automatically Decision Agent reasoning or an approval gate. No resource change Write (ITSM) Ticket create/update. Sanitized Write (Azure) Azure control-plane change. Approval required in production Write (Guest OS) Inside the VM, via a fixed-purpose runbook. Approval required Write (K8s/DevOps) Cluster or pipeline change. Approval required Validation Post-action measurement against defined thresholds Use case #1 — App Service: HTTP 500 after deployment The one you should build first. Clean trigger, bounded action, unambiguous validation. What happened Release 2026.03.12.4 was deployed to a production checkout App Service. The new code referenced an application setting that was never defined in the production slot. The application started fine — which is what makes this class of failure nasty — but every checkout operation that touched that setting returned HTTP 500. How Azure Monitor detected it Azure Monitor and Application Insights fired on a composite condition, not a single metric: HTTP 5xx rate above the operational threshold Failed availability tests Increased application exceptions Increased dependency failures Request latency above baseline P1 – Checkout App Service returning elevated HTTP 500 responses How the agent found the cause Identified the affected App Service and slot. Queried HTTP response and latency metrics. Queried Application Insights failed requests and exceptions. Identified the missing-setting exception as the dominant failure. Reviewed deployment history and Azure Activity Log. Correlated the error increase with release 2026.03.12.4 . Compared production against the previous deployment slot. Confirmed the previous slot remained healthy. Confirmed no matching Azure platform incident. Agent finding: HTTP 500 responses increased from 0.4% to 18% within three minutes of release 2026.03.12.4. Most failures reference a missing application setting. The previous deployment slot passes availability and dependency checks. Notice the shape of that sentence: a delta, a time correlation, and a known-good comparison. That's what makes it actionable rather than merely true. Agent action table # Action Type Detail 1 Investigate alert Read Reads HTTP 5xx rate, failed requests, latency, availability-test results 2 Analyze exceptions Read Identifies missing-configuration exception as dominant failure 3 Correlate deployment Read Finds release 2026.03.12.4 deployed three minutes before the spike 4 Compare slots Read Confirms previous slot healthy while production is failing 5 Rule out platform issue Read Checks Azure Resource Health and dependency health 6 Assess blast radius Read Determines checkout affected, unrelated services healthy 7 Create SNOW INC Write (ITSM) P1 against the Checkout CI, assigned to Application Operations 8 Update INC evidence Write (ITSM) Adds exceptions, deployment correlation, resource links, affected operations 9 Classify response Decision Classifies as deployment-caused; selects rollback 10 Pause release Write (DevOps) Pauses the failed release pipeline 11 Request approval Decision Requests approval to swap the previous healthy slot into production 12 Swap slot Write (Azure) Performs the approved swap only on the named App Service 13 Validate recovery Validation Confirms 5xx, latency, exceptions, dependencies, availability recover 14 Update/resolve INC Write (ITSM) Records approval, rollback, timestamps, recovery evidence 15 Create SNOW PRB Write (ITSM) Problem record for configuration-validation improvements Recovery note Production was reverted to the previous healthy deployment slot at 14:26 UTC. HTTP 500 responses declined from 18% to below 1%, and availability tests passed for 15 consecutive minutes. Preliminary cause: missing production configuration in release 2026.03.12.4. Try it yourself Break it: # Put a healthy build in the 'previous' slot first, then break production # by deploying code that reads an app setting which only exists in staging. az webapp config appsettings delete -g $RG -n app-checkout-demo \ --setting-names Checkout__PaymentProviderKey az webapp restart -g $RG -n app-checkout-demo Alert it: az monitor metrics alert create -g $RG -n "alert-checkout-5xx" \ --scopes "/subscriptions/$SUB/resourceGroups/$RG/providers/Microsoft.Web/sites/app-checkout-demo" \ --condition "total Http5xx > 20" \ --window-size 5m --evaluation-frequency 1m --severity 1 \ --description "P1 – Checkout App Service returning elevated HTTP 500 responses" Ask it (before the alert, to see the read phase in isolation): app-checkout-demo is returning HTTP 500s. Do not change anything. Investigate and tell me: 1. the dominant exception and its share of total failures 2. what deployed in the 30 minutes before the error rate changed 3. whether the 'previous' slot is healthy right now 4. whether Azure Resource Health shows a platform issue Give me the evidence chain and the smallest reversible mitigation. Expect it to propose: Proposed action: Swap slot 'previous' into production on app-checkout-demo Risk: brief connection drain (~10s). Fully reversible by swapping back. Validation: Http5xx < 1% and availability test passing for 15 minutes. [Approve] [Deny] Verify it: requests | where timestamp > ago(1h) | summarize failed = countif(success == false), total = count() by bin(timestamp, 1m) | extend failureRate = 100.0 * failed / total | render timechart 🎬 Best thing to record for a demo: the moment the Approve button appears with the evidence already attached. That single screen is the whole value proposition. Guardrails Scope write permissions to the named App Service, not the resource group. Verify the previous slot is healthy before swapping — a swap into a broken slot doubles the outage. Require human approval for production slot swaps. Preserve the failed deployment for analysis; don't let the pipeline overwrite it. Never grant subscription-level Contributor or Owner. Use case #2 — AKS: pods in CrashLoopBackOff What happened Image orders-api:4.18.0 referenced a configuration key that didn't exist in the production namespace. Eight of ten pods entered CrashLoopBackOff . The two survivors couldn't carry production traffic, producing latency and ingress 5xx. How Azure Monitor detected it Managed Prometheus / Container Insights detected: Unavailable replicas Increasing container restart count CrashLoopBackOff pod state Failed readiness and liveness checks Elevated ingress HTTP 5xx Reduced successful-request rate P1 – Orders API unavailable replicas in production AKS How the agent found the cause Identified the cluster, namespace, and deployment. Reviewed deployment availability and pod states. Examined logs from failing containers. Reviewed Kubernetes warning events. Compared current and previous ReplicaSets. Correlated the failure with revision 42. Identified the missing configuration key. Confirmed revision 41 was previously healthy. Checked node CPU, memory, storage, networking, status. Determined AKS infrastructure was healthy. Agent finding: Eight of ten Orders API pods entered CrashLoopBackOff during rollout of revision 42. Container logs show a missing configuration key. Revision 41, using image 4.17.6, was healthy. Step 10 is the one people skip. "Rule out the infrastructure" is what stops you from spending an hour on a node pool that was never the problem. Agent action table # Action Type Detail 1 Investigate workload Read Deployment status, unavailable replicas, readiness, restart counts 2 Analyze pod logs Read Startup failure from a missing configuration key 3 Analyze events Read Image pulls, mounts, scheduling, probes, container events 4 Correlate rollout Read Revision 42 deployed immediately before failure 5 Compare ReplicaSets Read Revision 41 was the previous healthy workload 6 Rule out infrastructure Read Nodes, memory, CPU, networking, storage healthy 7 Assess blast radius Read Eight of ten replicas unavailable 8 Create SNOW INC Write (ITSM) P1 against the Orders API CI 9 Update INC evidence Write (ITSM) Namespace, image, revision, errors, rollout correlation 10 Classify response Decision Application/configuration-caused 11 Request approval Decision Approval to pause and roll back revision 42 12 Pause failed rollout Write (K8s/DevOps) Prevents the release progressing or being reapplied 13 Roll back deployment Write (K8s) Rolls back only the named deployment to revision 41 14 Validate replicas Validation All expected replicas ready and stable 15 Validate application Validation Ingress errors decline; synthetic orders succeed 16 Update/resolve INC Write (ITSM) Revision, approval, rollback, recovery evidence 17 Create defect/task Write (ITSM) Add configuration validation to CI/CD Recovery note Orders API was rolled back from revision 42 to revision 41. All ten replicas are ready, restart counts have stabilized, ingress HTTP 5xx responses are below threshold, and synthetic order transactions are succeeding. Try it yourself Break it: az aks get-credentials -g $RG -n aks-demo kubectl create namespace orders kubectl create deployment orders-api -n orders --image=nginx:1.25 --replicas=10 kubectl rollout status deployment/orders-api -n orders # revision 41 equivalent, healthy # Now break it: point at an image whose entrypoint requires a missing env var kubectl set image deployment/orders-api -n orders orders-api=busybox:1.36 kubectl patch deployment orders-api -n orders --type=json -p='[ {"op":"add","path":"/spec/template/spec/containers/0/command", "value":["sh","-c","test -n \"$ORDERS_CONFIG_KEY\" || (echo \"FATAL: missing ORDERS_CONFIG_KEY\" >&2; exit 1); sleep 3600"]} ]' Watch it break: kubectl get pods -n orders -w kubectl rollout history deployment/orders-api -n orders Ask it: Pods in namespace 'orders' on aks-demo are crash looping. Read-only. Tell me: which revision introduced it, the exact container error, whether the node pool is healthy, and how many replicas are actually serving. Then tell me the last known-good revision and why you believe it was healthy. Expect it to propose: kubectl rollout undo deployment/orders-api -n orders --to-revision= scoped to that single deployment. Verify it: kubectl get deployment orders-api -n orders \ -o jsonpath='{.status.readyReplicas}/{.status.replicas}{"\n"}' kubectl get pods -n orders --no-headers | awk '{print $4}' | sort | uniq -c Guardrails Restrict access to the required cluster, namespace, and deployment. Do not grant cluster-admin . Require approval for rollback. Coordinate with GitOps reconciliation. If Flux or Argo owns that deployment, a kubectl rollout undo gets reverted within minutes and you've created a flapping outage. Either pause reconciliation first or roll back through Git. Do not permit namespace, persistent-volume, or cluster deletion. Use case #3 — Azure SQL Database: CPU saturation and timeouts The use case where the agent's job is to buy time, not to fix the problem. What happened A query execution plan changed after an application release. The new plan consumed substantially more CPU and workers. The database saturated, producing SQL dependency timeouts and failed customer requests. How Azure Monitor detected it Sustained CPU saturation Worker or session pressure Increased connection failures SQL dependency timeouts Query-duration deviation from baseline Degraded customer API success rate P1 – Customer API database saturation causing request timeouts How the agent found the cause Reviewed database CPU, workers, sessions, connections. Reviewed application SQL dependency failures. Checked for blocking and deadlocks. Ran approved read-only Query Store diagnostics. Identified the primary CPU-consuming query. Detected a recent execution-plan change. Correlated with an application release. Ruled out Azure service health and storage issues. Determined a temporary scale operation could restore service. Left permanent query remediation to the DBA team. Agent finding: Database CPU has remained saturated for 17 minutes. One query accounts for most recent CPU consumption and changed execution plan shortly before the incident. Application SQL dependency timeout rate is 23%. Step 10 is the design decision that makes this safe. The agent correctly diagnoses a plan regression and then deliberately does not fix it, because forcing a plan or dropping an index is a permanent, DBA-owned, change-controlled action. It buys capacity instead. Agent action table # Action Type Detail 1 Investigate database Read CPU, workers, sessions, connections, storage, availability 2 Analyze app impact Read SQL dependency latency, failures, affected API operations 3 Check blocking Read Approved diagnostics for blocking, deadlocks, connection growth 4 Analyze Query Store Read Highest-impact query and recent plan change 5 Correlate changes Read Recent application and database deployments 6 Rule out platform issue Read Service health, storage, database availability 7 Assess blast radius Read Affected applications; other databases healthy 8 Create SNOW INC Write (ITSM) P1 against the production database CI 9 Update INC evidence Write (ITSM) Utilization, query ID, timeouts, change correlation 10 Classify response Decision Scaling is mitigation; query changes remain DBA-owned 11 Calculate bounded scale Read Selects the smallest pre-approved capacity increase 12 Request approval Decision Database owner or incident commander 13 Scale database Write (Azure) Increases the affected database by one approved service step 14 Validate recovery Validation CPU, workers, timeouts, application success recover 15 Update INC Write (ITSM) Previous/new capacity, approval, timing, results 16 Create SNOW PRB Write (ITSM) Permanent query-remediation work 17 Create scale-down task Write (ITSM) Task/change to restore normal capacity after stability Recovery note Azure SQL capacity was temporarily increased by one approved service step. CPU declined from sustained saturation to 54%, and SQL dependency timeouts returned to baseline. Query Store indicates a probable execution-plan regression requiring permanent DBA remediation. Try it yourself Break it — generate a saturating workload against db-customer (an unindexed LIKE '%...%' scan in a tight loop from a container in the same region works fine on an S1). Alert it: az monitor metrics alert create -g $RG -n "alert-sql-cpu" \ --scopes "/subscriptions/$SUB/resourceGroups/$RG/providers/Microsoft.Sql/servers/sqlsrv-sre-demo/databases/db-customer" \ --condition "avg cpu_percent > 90" \ --window-size 5m --evaluation-frequency 1m --severity 1 \ --description "P1 – Customer API database saturation causing request timeouts" Ask it: db-customer is saturated. Read-only investigation. Identify the top CPU-consuming query, whether its plan changed recently, and what application release correlates. Do NOT propose index, schema, plan-forcing, or session-kill actions. Propose only the smallest temporary capacity step that restores service, and tell me what it costs per day and when we should scale back down. Read-only Query Store diagnostics the agent should run: SELECT TOP 10 qsq.query_id, qsp.plan_id, qsp.last_execution_time, SUM(qsrs.count_executions) AS executions, SUM(qsrs.avg_cpu_time * qsrs.count_executions) AS total_cpu_us FROM sys.query_store_query AS qsq JOIN sys.query_store_plan AS qsp ON qsp.query_id = qsq.query_id JOIN sys.query_store_runtime_stats AS qsrs ON qsrs.plan_id = qsp.plan_id JOIN sys.query_store_runtime_stats_interval AS qsrsi ON qsrsi.runtime_stats_interval_id = qsrs.runtime_stats_interval_id WHERE qsrsi.start_time > DATEADD(hour, -2, GETUTCDATE()) GROUP BY qsq.query_id, qsp.plan_id, qsp.last_execution_time ORDER BY total_cpu_us DESC; Expect it to propose: az sql db update -g $RG -s sqlsrv-sre-demo -n db-customer --service-objective S2 — exactly one step, on exactly that database. Guardrails Restrict scaling to the named database. Define minimum and maximum capacity. Require approval for scale-up and scale-down. Time-limit temporary capacity — an un-reversed emergency scale-up is how a P1 becomes a budget incident. Do not autonomously force plans, terminate sessions, modify indexes, or change schema. Track the temporary cost impact with an Azure Cost Management alert. Use case #4 — Azure Cosmos DB: HTTP 429 throttling The one where the correct root cause is "we're succeeding." What happened A marketing campaign increased Product Catalog traffic by roughly 40%. The Cosmos DB container hit its provisioned throughput ceiling. HTTP 429s increased, and client retries amplified application latency. How Azure Monitor detected it High normalized RU consumption Increased HTTP 429 responses Elevated server-side latency Application dependency failures Sustained operation near the throughput limit P2 – Cosmos DB throttling affecting Product Catalog requests How the agent found the cause Reviewed normalized RU consumption and throttled requests. Identified the affected database and container. Reviewed regional and partition behavior. Checked for hot-partition evidence. Reviewed application retry telemetry. Compared current traffic with the historical baseline. Correlated demand with the marketing campaign. Checked recent application deployments. Checked Azure service health. Determined the primary cause was legitimate demand. Agent finding: The Product Catalog container is at its configured throughput ceiling, and 21% of requests are being throttled. Traffic increased by approximately 40% following a campaign launch. No deployment or regional platform issue correlates with the event. Step 4 is the fork in the road. If consumption is uneven across partitions, more RU/s is money set on fire — the correct answer is an architecture change, not a scale-up. The agent has to check before it recommends. Agent action table # Action Type Detail 1 Investigate throttling Read RU consumption, 429 rate, latency, requests, availability 2 Identify scope Read Account, database, container, operations, regions 3 Analyze demand Read Compares traffic and RU consumption with historical patterns 4 Check partitions Read Looks for uneven partition consumption where telemetry permits 5 Analyze retries Read Determines whether client retries are amplifying the incident 6 Correlate events Read Links demand to campaign traffic; excludes release/platform issues 7 Assess blast radius Read Affected catalog operations; unaffected containers 8 Create SNOW INC Write (ITSM) P2 against the Product Catalog CI 9 Update INC evidence Write (ITSM) RU, throttling, latency, traffic, partition evidence 10 Classify response Decision Demand-driven unless hot-partition evidence exists 11 Calculate throughput Read Smallest increase within the cost ceiling 12 Request approval Decision Approval for a temporary throughput increase 13 Increase throughput Write (Azure) Raises throughput only to the approved maximum 14 Validate recovery Validation 429 rate and latency recover 15 Update/resolve INC Write (ITSM) Throughput, approval, cost implication, recovery 16 Create capacity task Write (ITSM) Work to return throughput to normal 17 Create SNOW PRB Write (ITSM) Partition or retry improvements, if required Recovery note Provisioned throughput was increased within the approved production limit. HTTP 429 responses declined from 21% to below 1%, and Product Catalog latency returned to baseline. The increase is temporary and will be reviewed after campaign traffic subsides. Try it yourself Break it: the container was created at 400 RU/s. Drive a few hundred reads per second at it and you'll be throttled within seconds. Alert it: az monitor metrics alert create -g $RG -n "alert-cosmos-429" \ --scopes "/subscriptions/$SUB/resourceGroups/$RG/providers/Microsoft.DocumentDB/databaseAccounts/cosmos-sre-demo" \ --condition "total TotalRequests where StatusCode == 429 > 100" \ --window-size 5m --evaluation-frequency 1m --severity 2 \ --description "P2 – Cosmos DB throttling affecting Product Catalog requests" Ask it: cosmos-sre-demo container 'products' is throttling. Read-only. Before recommending anything, tell me whether RU consumption is EVEN across physical partitions or concentrated. If it is concentrated, do not recommend a throughput increase — recommend an architecture problem record instead. If it is even, tell me the smallest RU/s that clears throttling and the daily cost delta. That prompt is the whole use case. Getting the agent to refuse the easy answer under a stated condition is the skill. Verify it: AzureDiagnostics | where ResourceProvider == "MICROSOFT.DOCUMENTDB" | where Category == "DataPlaneRequests" | summarize throttled = countif(statusCode_s == "429"), total = count() by bin(TimeGenerated, 1m) | extend throttleRate = 100.0 * throttled / total | render timechart Guardrails Define a maximum throughput ceiling the agent may not exceed. Restrict changes to the named container. Require approval for increases and reductions. Do not permit deletion, consistency-level changes, or region changes. Create cost alerts for prolonged increased throughput. Treat persistent hot partitions as an architecture issue, never as a scaling issue. Use case #5 — Azure VM: OS/root disk full The most operationally dangerous use case in this post, and the one with the most interesting guardrail design. What happened A legacy payment application generated excessive trace logs. Log rotation stopped working and the root filesystem filled. The VM stayed available at the Azure platform layer — heartbeat green, Resource Health fine — but the application stopped, because it could no longer write to disk. This is the classic "green dashboard, dead service" failure. Platform-layer monitoring alone will never catch it. How Azure Monitor detected it Azure Monitor Agent and guest telemetry detected: Critically low filesystem free space Rapid filesystem consumption Application process stopped Failed availability tests Elevated HTTP 5xx Healthy VM heartbeat but unhealthy application P1 – Legacy payment application unavailable due to full VM OS disk How the agent found the cause Confirmed the VM was online. Confirmed Azure Monitor Agent heartbeat. Identified the affected root/OS filesystem. Reviewed the free-space trend. Correlated application failure with disk exhaustion. Found 86 GB of growth in the application trace directory. Reviewed recent deployments and logging changes. Checked log-rotation status. Confirmed the files matched the approved cleanup policy. Excluded customer data, database files, audit logs, and system files. Agent finding: The VM is healthy at the Azure platform layer, but the OS volume has less than 1% free space. The approved application trace directory grew by 86 GB in six hours. The payment service stopped when it could no longer write to disk. Agent action table # Action Type Detail 1 Investigate VM Read VM, agent heartbeat, Azure platform health 2 Analyze filesystem Read Root volume and free-space trend 3 Correlate app failure Read Service stopped after disk exhaustion 4 Identify disk consumer Read Abnormal growth in an approved trace directory 5 Check changes Read Deployments, logging changes, rotation, scheduled tasks 6 Validate cleanup scope Read Candidate files meet approved path, type, and age rules 7 Protect data Decision Excludes databases, customer data, security logs, unknown files 8 Create SNOW INC Write (ITSM) P1 against the payment VM/application CI 9 Update INC evidence Write (ITSM) Disk usage, growth timeline, service impact, cleanup scope 10 Request approval Decision Approval for the restricted recovery runbook 11 Archive logs Write (Guest OS) Archives eligible files to protected Azure Storage 12 Remove eligible files Write (Guest OS) Removes only successfully archived, allowlisted files 13 Run log rotation Write (Guest OS) Executes the approved rotation operation 14 Restart service Write (Guest OS) Restarts only the named payment service if necessary 15 Validate recovery Validation Disk, application, archive, availability, growth stabilization 16 Update/resolve INC Write (ITSM) Bytes processed, exclusions, approval, results 17 Create SNOW PRB Write (ITSM) Permanent logging and rotation remediation Recovery note The approved recovery runbook archived and removed 82 GB of eligible application trace files. OS-volume free space is now 31%. The payment service was restarted and has passed health checks for 15 minutes. No database, customer, security, or system files were modified. That last sentence is not decoration. It is the sentence your auditor will read. Try it yourself Break it (lab VM only — this fills the root disk): az vm run-command invoke -g $RG -n vm-payments-linux \ --command-id RunShellScript --scripts " mkdir -p /var/log/payments/trace fallocate -l 24G /var/log/payments/trace/trace-$(date +%s).log df -h / " Ask it: vm-payments-linux root filesystem is nearly full and the payment service is down. Read-only first. Tell me: - exactly which directory grew, by how much, over what window - whether log rotation is configured and when it last ran - what changed in the last 24 hours that would explain it Then tell me which files are inside the approved cleanup allowlist (/var/log/payments/trace/*.log, older than 2h) and which are NOT, and confirm no database, audit, or customer data files are in scope. Do not delete anything. The critical design point. SRE Agent blocks delete and remove commands outright. You cannot have it rm those files through its Azure CLI surface, and you should be glad. The correct implementation is a fixed-purpose, version-controlled runbook that the agent invokes with tightly bounded parameters: # What the agent is allowed to call — one runbook, allowlisted parameters, nothing else az automation runbook start \ -g $RG --automation-account-name aa-sre-runbooks \ -n "Reclaim-TraceDiskSpace" \ --parameters vmName=vm-payments-linux \ allowedPath=/var/log/payments/trace \ pattern='*.log' \ minAgeHours=2 \ maxBytes=90000000000 \ archiveToContainer=payments-trace-archive \ requireArchiveBeforeDelete=true \ dryRun=false The runbook — not the agent — owns the destructive logic, and it enforces: path allowlist (refuse anything outside allowedPath ) filename pattern allowlist minimum file age maximum total bytes per execution successful archive verified before any deletion hard-refuse if any candidate file is unknown, or matches a protected pattern ( *.mdf , *.bak , /var/log/audit/* , *.key , *.pem ) a cooldown that prevents re-execution within N hours Verify it: az vm run-command invoke -g $RG -n vm-payments-linux \ --command-id RunShellScript \ --scripts "df -h /; systemctl is-active payments.service; ls -la /var/log/payments/trace | head" Guardrails Do not give SRE Agent unrestricted SSH or shell access. Ever. This is the single highest-leverage rule in this post. Use a version-controlled, fixed-purpose runbook. Allowlist paths, patterns, file ages, and maximum cleanup size. Stop if the responsible files are unknown. A disk filled by something you can't identify is a security event until proven otherwise. Require successful archival before deletion. Prevent repeated execution with a cooldown. Treat cleanup as temporary mitigation — the problem record is the fix. Use case #6 — Linux VM: anomalous CPU saturation Where "anomalous" is doing all the work. What happened Release 7.3.1 introduced an immediate retry loop when an inventory dependency failed. The application retried continuously without backoff, consuming nearly all VM CPU and causing request timeouts. How Azure Monitor detected it The alert deliberately combined conditions rather than firing on a threshold: CPU significantly outside the historical baseline Persistent saturation over multiple evaluations Increased request latency Increased dependency failures No approved maintenance or batch workload active P1 – Production order-processing Linux VM experiencing anomalous CPU saturation A static "CPU > 90%" rule on a batch-processing VM is a pager that everyone learns to ignore. The composite condition is what makes the alert worth waking someone — or an agent — for. How the agent found the cause Confirmed the VM was online. Compared current CPU with historical behavior. Identified the order-processing service as the main CPU consumer. Reviewed application request latency. Reviewed downstream dependency failures. Analyzed application retry logs. Correlated CPU growth with release 7.3.1. Excluded expected batch jobs. Excluded Azure maintenance or platform issues. Confirmed other application instances had sufficient capacity. Agent finding: CPU increased from a normal range of 35–50% to 98% four minutes after deployment 7.3.1. The order-processing service is repeatedly calling a failed dependency without backoff. No expected batch job or platform maintenance is active. Agent action table # Action Type Detail 1 Confirm anomaly Read Compares current CPU with the historical baseline 2 Analyze impact Read Latency, timeouts, availability, dependencies 3 Identify process Read Named order service is the primary CPU consumer 4 Correlate deployment Read Release 7.3.1, four minutes before saturation 5 Analyze logs Read Dependency retry loop without backoff 6 Rule out expected work Read Excludes batch, backup, maintenance, scheduled processing 7 Assess blast radius Read Traffic and available capacity on other instances 8 Create SNOW INC Write (ITSM) P1 against the Order Processing CI 9 Update INC evidence Write (ITSM) CPU, process, dependency, release, customer impact 10 Classify response Decision Rollback, not VM resize or arbitrary process termination 11 Request approval Decision Approval to drain, roll back, restart the service 12 Drain VM Write (Azure) Removes the VM from load-balancer rotation 13 Restore prior release Write (Guest OS/DevOps) Restores known-good version or configuration 14 Restart named service Write (Guest OS) Restarts only orders-service 15 Validate recovery Validation CPU, retries, latency, dependencies, health recover 16 Return to rotation Write (Azure) Restores traffic only after successful validation 17 Update/resolve INC Write (ITSM) Drain, rollback, restart, approval, results 18 Create defect Write (ITSM) Retry, backoff, and circuit-breaker remediation Steps 12 and 16 are the pattern to steal: drain before you touch, restore traffic only after validation passes. Most homegrown automation restarts a service while it's still taking traffic and turns a degradation into an outage. Recovery note Release 7.3.1 introduced a retry loop when the inventory dependency failed. After approval, the VM was drained, the previous release was restored, and orders-service was restarted. CPU declined from 98% to 43%, and application latency returned to baseline. Try it yourself Break it: az vm run-command invoke -g $RG -n vm-payments-linux \ --command-id RunShellScript --scripts " nohup bash -c 'while true; do curl -s -m 1 http://127.0.0.1:9/inventory >/dev/null 2>&1; done' & nohup bash -c 'while true; do :; done' & echo started " Ask it: CPU on vm-payments-linux is at 98%. Read-only. First: is this actually anomalous, or is it consistent with this VM's historical pattern for this hour and day of week? Show me the baseline. If anomalous: which process, which dependency is it calling, at what rate, and what deployed immediately before? Do NOT propose killing the top process or resizing the VM. Expect it to propose a drain → restore → restart sequence, with the drain step as a separate approval from the restart. Guardrails Do not automatically terminate the highest-CPU process. It is very often a legitimate workload, and occasionally it's a security incident you just destroyed the evidence for. Allow operations only for named services. Drain before service restart where possible. Escalate unknown or suspicious processes to security rather than remediating them. Limit restart attempts. Do not permanently resize the VM when the evidence points to faulty code. Resizing to survive a retry storm is buying hardware to host a bug. Use case #7 — Windows VM with IIS: memory leak What happened Release ClaimsPortal 5.9.0 introduced a memory leak in the Claims IIS application pool. Memory consumption climbed over several hours, paging began, request queues grew, and IIS returned HTTP 503. How Azure Monitor detected it Azure Monitor Agent collected available memory, committed bytes, paging activity, process working set, IIS request queues, HTTP 500/503 responses, and availability-test results. The alert required sustained abnormal growth, not a brief spike. P1 – Memory exhaustion affecting production IIS application How the agent found the cause Confirmed the VM and monitoring agent were healthy. Compared memory behavior with the historical baseline. Identified the relevant w3wp.exe process. Mapped it to the Claims application pool. Reviewed paging and request queues. Correlated HTTP 503 errors with low available memory. Reviewed Windows Event Logs. Correlated the growth with release 5.9.0. Excluded antivirus and scheduled-reporting activity. Confirmed another instance could carry traffic. Agent finding: Available memory declined from 42% to 4% over three hours. The Claims application pool grew from 1.8 GB to 11.6 GB without releasing memory after traffic normalized. Paging and HTTP 503 errors followed. The pattern began after release 5.9.0. "Without releasing memory after traffic normalized" is the sentence that distinguishes a leak from load. Cache growth under load is normal; failure to return afterwards is not. Agent action table # Action Type Detail 1 Confirm anomaly Read Memory, committed bytes, paging vs. baseline 2 Identify process Read Maps growing w3wp.exe to the Claims pool 3 Analyze IIS health Read Pools, queues, HTTP errors, availability 4 Correlate release Read Links sustained growth to release 5.9.0 5 Rule out other causes Read Excludes scheduled jobs, antivirus, maintenance 6 Assess capacity Read Confirms another instance can serve traffic 7 Create SNOW INC Write (ITSM) P1 against the Claims Portal CI 8 Update INC evidence Write (ITSM) Memory, paging, pool, release, HTTP impact 9 Classify response Decision Targeted recycling, not a full VM restart 10 Request approval Decision Approval to drain and recycle the named pool 11 Drain VM Write (Azure) Removes VM from load-balancer rotation 12 Capture diagnostics Write (Guest OS) Captures approved diagnostics to a protected location 13 Recycle app pool Write (Guest OS) Recycles only the Claims application pool 14 Validate recovery Validation Memory, paging, queues, HTTP errors, availability recover 15 Return to rotation Write (Azure) Restores traffic after health checks pass 16 Update/resolve INC Write (ITSM) Memory before/after, approval, stability 17 Create SNOW PRB Write (ITSM) Memory-leak remediation Step 12 before step 13 matters: recycling the pool destroys the evidence. Capture first. Recovery note Abnormal memory growth was isolated to the Claims application pool following release 5.9.0. The VM was drained, the approved application pool was recycled, and health checks passed before traffic was restored. Available memory increased from 4% to 61%, and HTTP 503 responses stopped. Try it yourself Break it (lab VM only): az vm run-command invoke -g $RG -n vm-claims-win ` --command-id RunPowerShellScript --scripts " Install-WindowsFeature Web-Server -IncludeManagementTools New-WebAppPool -Name 'ClaimsPool' # Simulate the leak \$leak = New-Object System.Collections.ArrayList 1..40 | ForEach-Object { [void]\$leak.Add((New-Object byte[] 100MB)) ; Start-Sleep -Milliseconds 200 } " Ask it: vm-claims-win available memory is at 4%. Read-only. Map the growing process to an IIS application pool. Show me the memory curve for the last 6 hours and tell me whether memory was released after traffic dropped. Correlate with deployment history. Then propose the most targeted possible mitigation — I do not want a VM restart. Expect it to propose: Restart-WebAppPool -Name "ClaimsPool" …and nothing else on that machine. Guardrails Recycle only the named application pool. Avoid full VM restart as the first response — it's a bigger hammer with a longer outage and it destroys the leak evidence. Do not copy memory dumps into ServiceNow. They contain credentials, tokens, and customer data. Store diagnostics in a secured location; put the link in the ticket. Prevent repeated automatic recycling — a pool that needs recycling every 40 minutes is an incident, not a routine. Escalate if the leak returns during the observation period. Use case #8 — Virtual Machine Scale Set: unhealthy instance The most autonomy-ready use case in the list, and the reason is stateless workloads. What happened A configuration extension failed while VMSS instance 17 was being provisioned. The VM was running but its application service never started. The instance failed application health probes and caused intermittent errors. How Azure Monitor detected it Reduced healthy backend count Application Health extension failure Backend health-probe failure VM extension provisioning failure Instance-specific errors Elevated intermittent HTTP 5xx P2 – Unhealthy VM Scale Set instance causing intermittent API failures How the agent found the cause Reviewed the VMSS healthy-instance count. Identified instance 17 as the only unhealthy instance. Reviewed backend health. Compared instance 17 with healthy instances. Checked the image and VMSS model. Reviewed VM extension state. Found the configuration extension failure. Reviewed boot and application diagnostics. Confirmed the workload was stateless. Confirmed eleven instances could carry production traffic. Agent finding: Instance 17 is the only unhealthy member of the 12-instance scale set. Its application health probe has failed since 09:18 UTC. The configuration extension failed during provisioning, and the application service never started. Eleven healthy instances can maintain service. Agent action table # Action Type Detail 1 Investigate VMSS Read Instance health, provisioning state, healthy count 2 Identify instance Read Instance 17 is the only unhealthy member 3 Analyze backend health Read Confirms the instance fails application probes 4 Compare instances Read Image, model, extensions, configuration 5 Find extension failure Read Locates the failed configuration extension 6 Review diagnostics Read Boot, extension, and application diagnostics 7 Assess safe capacity Read Eleven instances can carry traffic 8 Confirm statelessness Decision Replacement won't destroy required local state 9 Create SNOW INC Write (ITSM) P2 against the VMSS/application CI 10 Update INC evidence Write (ITSM) Instance, extension error, health, capacity evidence 11 Classify response Decision Replacement or reimage per approved procedure 12 Request approval Decision Approval to isolate and replace instance 17 13 Isolate instance Write (Azure) Ensures the instance receives no production traffic 14 Preserve evidence Read/Write Stores approved diagnostic evidence securely 15 Replace instance Write (Azure) Reimages or replaces only instance 17 16 Validate provisioning Validation Image, model, and extensions deploy successfully 17 Validate service Validation Backend health, capacity, customer errors recover 18 Update/resolve INC Write (ITSM) Replacement, approval, diagnostics, recovery 19 Create SNOW PRB Write (ITSM) Extension and image-validation improvement work Steps 8 and 7 are the gate. Replacing a stateless instance with eleven healthy peers is genuinely low risk. Replacing a stateful instance, or replacing one when you're already at minimum capacity, is an outage. Both must be confirmed, not assumed. Recovery note VMSS instance 17 was isolated after its configuration extension failed and the application service did not start. Diagnostic evidence was captured, and the instance was replaced after approval. The replacement passed extension, application, and backend health checks. The scale set has returned to 12 healthy instances. Try it yourself Break it: INSTANCE_ID=$(az vmss list-instances -g $RG -n vmss-api-demo \ --query "[5].instanceId" -o tsv) az vmss extension set -g $RG --vmss-name vmss-api-demo \ --name CustomScript --publisher Microsoft.Azure.Extensions \ --settings '{"commandToExecute":"exit 1"}' Ask it: vmss-api-demo has an unhealthy instance. Read-only. Identify which instance, since when, and the specific extension error. Confirm for me: (a) the workload is stateless, (b) how many healthy instances remain, and (c) whether remaining capacity can carry current traffic with 20% headroom. Only if all three are satisfied, propose a reimage of that single instance. Capture diagnostics before proposing anything. Expect it to propose: az vmss reimage -g $RG -n vmss-api-demo --instance-id $INSTANCE_ID Verify it: az vmss get-instance-view -g $RG -n vmss-api-demo --instance-id $INSTANCE_ID \ --query "vmHealth.status.code" az vmss list-instances -g $RG -n vmss-api-demo -o table Guardrails Confirm the workload is stateless before any replacement. Confirm sufficient healthy capacity first. Preserve diagnostic evidence before replacement — the instance is your only copy of the failure. Restrict permissions to the named VMSS. Limit simultaneous replacements. One at a time. An agent that reimages six instances because six probes failed has just caused the outage it was investigating. Do not permit deletion of the entire scale set. Use case #9 — Application Gateway: HTTP 502 from unhealthy backends The cross-component change nobody coordinated. What happened Customer Portal release 9.2 changed the backend listener from port 443 to 8443. Application Gateway remained configured to connect on 443. All backend probes failed and customers received HTTP 502. How Azure Monitor detected it Increased Application Gateway HTTP 502 responses Increased failed requests Four unhealthy backends Reduced healthy-host count Failed synthetic availability tests P1 – Application Gateway returning HTTP 502 due to unhealthy backends How the agent found the cause Reviewed Application Gateway metrics. Retrieved backend health. Identified the affected pool. Reviewed probe path, protocol, host header, and port. Reviewed backend settings. Confirmed the application responded on 8443. Confirmed the gateway used 443. Correlated the mismatch with release 9.2. Reviewed NSG, route, DNS, certificate, and WAF changes. Excluded networking, certificate, and platform-health issues. Agent finding: HTTP 502 responses began at 16:07 UTC. All four Customer Portal backends are unhealthy. Release 9.2 changed the backend listener to port 8443, while Application Gateway continues to use port 443. No NSG, routing, or certificate issue correlates with the incident. Step 9 is what separates a real investigation from a lucky guess. A 502 has at least six plausible causes — NSG, UDR, DNS, expired cert, WAF rule, backend down. The agent has to exclude them, in writing, before you trust the conclusion. Agent action table # Action Type Detail 1 Investigate gateway Read HTTP 502, failed requests, latency, backend counts 2 Inspect backend health Read Identifies four unhealthy Customer Portal backends 3 Review configuration Read Settings, probes, protocol, port, TLS, routing 4 Test backend state Read App responds on 8443 while gateway uses 443 5 Correlate changes Read Links mismatch to Customer Portal release 9.2 6 Rule out networking Read NSGs, routes, DNS, TLS, WAF, platform health 7 Assess blast radius Read Confirms the Customer Portal pool is unavailable 8 Create SNOW INC Write (ITSM) P1 against the portal/gateway CI 9 Update INC evidence Write (ITSM) 502s, backend, port, deployment, known-good configuration 10 Classify response Decision Restore the known-good backend listener 11 Request approval Decision Approval to restore source-controlled configuration 12 Restore listener Write (Application) Restores the application listener to approved port 443 13 Validate backend health Validation All four backends become healthy 14 Validate application Validation HTTP 502 declines; synthetic transactions pass 15 Verify controls Read Confirms no WAF, TLS, routing, or NSG control was weakened 16 Update/resolve INC Write (ITSM) Cause, restoration, approval, recovery 17 Create SNOW CHG Write (ITSM) Coordinated change for the intended port migration 18 Create SNOW PRB Write (ITSM) Cross-component deployment-validation work Step 12 is a genuinely important choice. There were two ways to fix this: change the application back to 443, or change the gateway to 8443. The agent restores the application to the known-good, source-controlled state rather than mutating the gateway to match an unapproved change. One of those is a rollback; the other is ratifying an unreviewed change during an outage. Then step 17 files a proper change record for the migration the team clearly intended to do. Step 15 exists because the fastest way to make a 502 disappear is to disable TLS validation. The agent must prove it didn't take the fast way. Recovery note Customer Portal release 9.2 changed the backend listener from port 443 to 8443 without a coordinated gateway change. The application listener was restored to the previous configuration. All four backends are healthy, HTTP 502 responses returned to baseline, and synthetic login tests passed for 15 minutes. Try it yourself Ask it: appgw-portal is returning 502s and all backends are unhealthy. Read-only. Walk me through the exclusion, explicitly, for each of: NSG, UDR/route table, DNS resolution, backend TLS certificate, WAF rule blocking, backend process down, and probe configuration mismatch. State which you ruled out and the evidence for each. Then tell me the known-good configuration and where it is source-controlled. Verify it: az network application-gateway show-backend-health \ -g $RG -n appgw-portal \ --query "backendAddressPools[].backendHttpSettingsCollection[].servers[].{addr:address,health:health}" -o table Guardrails Do not disable TLS validation. Not to test, not temporarily, not "just to confirm." Do not weaken NSGs or WAF policies as a mitigation. Use source-controlled configuration as the definition of "known-good." Restore a known-good state during the incident; migrate through a change record afterwards. Require approval for gateway or backend changes. Validate full application transactions, not just health probes. A probe returning 200 on /health proves very little. Use case #10 — Azure Service Bus: queue and dead-letter backlog The one that teaches the most important lesson in the entire post. What happened Release fulfillment-worker:6.4.0 couldn't deserialize messages containing a new deliveryWindow field. Consumer throughput dropped by 92%. The active backlog grew rapidly, and incompatible messages entered the dead-letter queue. How Azure Monitor detected it Increasing active-message count Increasing oldest-message age Dead-letter growth Reduced completed-message rate Consumer application errors Delayed downstream business processing P1 – Production order-event backlog delaying fulfillment How the agent found the cause Reviewed active, incoming, outgoing, and dead-letter counts. Calculated message arrival and completion rates. Confirmed the backlog was growing. Reviewed consumer instance health. Reviewed consumer errors and restarts. Checked downstream dependency health. Checked Service Bus authentication and authorization. Correlated the throughput decline with release 6.4.0. Found deserialization errors for the new field. Determined scaling more broken consumers would amplify failures. Agent finding: The order-events queue grew from 3,000 to 185,000 active messages in 35 minutes. Consumer throughput dropped by 92% immediately after release fulfillment-worker:6.4.0. Application logs show deserialization failures involving the new deliveryWindow field. Step 10 is the whole reason to use a reasoning agent instead of an autoscale rule. Every metric here screams "scale out the consumers." An HPA would have done exactly that, and every new replica would have dead-lettered messages faster. Agent action table # Action Type Detail 1 Investigate queue Read Active, incoming, outgoing, scheduled, DLQ counts 2 Calculate flow rates Read Confirms arrival exceeds completion; backlog growing 3 Inspect consumers Read Health, instance count, scaling, errors, dependencies 4 Analyze failures Read Deserialization exceptions involving the new field 5 Correlate release Read Links the 92% throughput reduction to release 6.4.0 6 Rule out Service Bus Read Health, authorization, throttling, networking, service status 7 Assess blast radius Read Backlog age and fulfillment impact 8 Protect message data Decision Prevents payloads or personal data entering ServiceNow 9 Create SNOW INC Write (ITSM) P1 against the fulfillment integration CI 10 Update INC evidence Write (ITSM) Backlog, age, flow, exception, release, business impact 11 Classify response Decision Rollback, not scaling broken consumers 12 Request approval Decision Approval to restore consumer 6.3.7 13 Roll back consumer Write (Azure/K8s) Rolls back through the approved deployment platform 14 Restore capacity Write (Azure/K8s) Restores approved consumer instance count 15 Validate processing Validation Consumer and downstream health 16 Validate backlog Validation Completion exceeds arrival; DLQ growth stops 17 Update INC Write (ITSM) Rollback, rates, estimated drain time, approval 18 Maintain incident Decision Keeps the incident open until backlog age meets the objective 19 Create SNOW CHG Write (ITSM) Separate controlled change for DLQ replay 20 Create SNOW PRB Write (ITSM) Message-contract compatibility remediation Recovery note Consumer release 6.4.0 could not deserialize messages containing the new deliveryWindow field. The fulfillment worker was rolled back to 6.3.7. Consumer throughput recovered, new dead-letter growth stopped, and the active backlog is draining at approximately 7,500 messages per minute. Dead-letter replay requires a separately approved procedure. The lesson: technical recovery is not business recovery Step 18 is the most important row in this entire post. At the moment of rollback, every technical signal is green. Consumers are healthy. Throughput has recovered. The DLQ has stopped growing. An agent optimizing for metrics would resolve the incident right there and go back to sleep. But there are still 185,000 unshipped orders and a dead-letter queue full of messages that need a separately approved replay procedure. Customers are still affected. The incident stays open until backlog age meets the business objective, not until the graphs look nice. Encode this in the response plan explicitly: Do not resolve this incident when consumer health recovers. Resolution criteria: 1. Completion rate exceeds arrival rate for 15 consecutive minutes, AND 2. Oldest active message age is under 5 minutes, AND 3. Dead-letter count has not increased for 30 minutes. DLQ replay is out of scope for this incident. File a separate change record. Try it yourself Break it: # Flood the queue while no consumer is running for i in $(seq 1 5000); do az servicebus queue message send -g $RG --namespace-name sb-sre-demo \ -q order-events --body "{\"orderId\":$i,\"deliveryWindow\":\"2026-08-09T10:00Z\"}" 2>/dev/null done az servicebus queue show -g $RG --namespace-name sb-sre-demo -n order-events \ --query "countDetails" -o json Ask it: order-events on sb-sre-demo has a growing backlog. Read-only. Give me: arrival rate, completion rate, current active count, oldest message age, DLQ count and DLQ growth rate, and the projected drain time at current rates. Then tell me why scaling out consumers is or is not the correct action here. Do not include any message payloads or customer data in your answer. That last line is not optional. Message bodies routinely contain names, addresses, and payment references — and everything the agent writes goes into a ticket. Verify it: az servicebus queue show -g $RG --namespace-name sb-sre-demo -n order-events \ --query "{active:countDetails.activeMessageCount, dlq:countDetails.deadLetterMessageCount}" -o json Guardrails Never automatically purge queues. Do not automatically replay dead-letter messages. Replay without idempotency guarantees means duplicate charges and duplicate shipments. Do not place message payloads in ServiceNow. Confirm idempotency before replay. Scale consumers only when the processing path is healthy. Require separate approval for replay or queue configuration changes. Keep the incident open until business recovery is confirmed. 7. The ITSM integration model Recommended incident fields Field Example Short description Production Orders API pods failing after deployment Configuration item prod-aks-orders-api Assignment group Container Platform Operations Impact High Urgency High Environment Production Azure resource ID Full affected Azure resource ID Azure alert ID Azure Monitor alert correlation identifier SRE investigation Link to the SRE Agent investigation thread Current impact Eight of ten replicas unavailable Probable cause Missing configuration in latest release Confidence High Proposed action Roll back to revision 41 Approval Approver and UTC timestamp Validation Ten replicas ready and synthetic tests passing Resolution Service restored through rollback The Confidence field earns its place. An agent that says "probable cause: missing configuration (confidence: low)" is far more useful than one that always sounds certain, because it tells the human how much to verify before approving. Fields the agent can set directly (preview): assignment_group , category , subcategory , impact , urgency , priority , short_description , and any custom u_* field. It cannot change incident state through field updates — acknowledge and resolve are separate, dedicated tools. Recommended agent-generated timeline Every work note should carry: Element Example Timestamp 2026-03-18 14:26 UTC Observation HTTP 500 rate increased to 18% Evidence Link to the Application Insights query Change correlation Incident started three minutes after deployment Probable cause Missing production application setting Confidence High Proposed action Swap to previous healthy slot Risk Temporary deployment rollback Approval Incident commander and timestamp Execution Slot-swap operation and result Validation 5xx below 1% for 15 minutes Follow-up Problem record for configuration validation Note that Evidence is a link, not a paste. This is a deliberate data-protection pattern: the ticket carries a pointer to the query, and the query results stay in the system that already has the right access controls. Deduplication strategy Do not create one incident per alert, pod, queue, or VMSS instance. Correlate on: application/business service + Azure resource ID + environment (production) + alert-rule family + active incident time window Related alerts attach to the existing incident as evidence or child alerts. Azure Monitor already merges recurring alerts into a single thread when it's the bound platform; for ServiceNow, this correlation key is yours to implement. Get this wrong and your first AKS incident produces eight incidents, eight investigations, and eight rollback proposals for the same deployment. Record responsibilities Record Purpose Example Incident (INC) Restore service quickly and safely App Service HTTP 500 outage Problem (PRB) Identify and remove the underlying cause Missing deployment configuration validation Change (CHG) Govern permanent or higher-risk production changes Coordinated Application Gateway port migration Engineering defect/task Correct application or automation behavior Add retry backoff to the Linux application The agent must clearly distinguish temporary mitigation from permanent correction. Every single use case above ends with a follow-up record, and that's not bureaucratic theatre — an agent that mitigates flawlessly and never files a problem record is an agent that lets the same outage recur forever while making the metrics look great. 8. Reality check: where you have to build This is the section that will save you a month. The PDF this post is built from is explicit that these are target response patterns, not guaranteed zero-configuration behavior. Having now checked each pattern against the product documentation, here is exactly where the gaps are and what fills them. # The pattern assumes What's actually documented What you must build 1 Agent creates a ServiceNow INC ServiceNow is an inbound platform. Documented writes: post discussion entries, acknowledge, resolve, plus field updates (preview) Incidents should originate in ServiceNow (via its own Azure Monitor integration) and flow in. If you truly need agent-initiated creation, add a Python tool or MCP server against the ServiceNow Table API 2 Agent creates PRB / CHG / defect records Not a documented first-class ServiceNow action Same: a custom tool against /api/now/table/problem and /change_request . This is ~30 lines of Python and worth doing properly, with a least-privileged integration user 3 ServiceNow and PagerDuty both connected Only one incident platform active at a time; switching disconnects the other Bind the agent to your system of record (ServiceNow). Reach the pager through a connector, Teams/Slack, or a webhook 4 Agent runs guest-OS cleanup ( rm , rotate, restart) delete and remove commands are blocked outright. az keyvault blocked. Management locks respected Wrap all guest-OS work in fixed-purpose Azure Automation runbooks or a constrained az vm run-command script, invoked with allowlisted parameters. See use case #5 5 Agent "pauses the release pipeline" Requires the GitHub or Azure DevOps connector, plus permissions on that pipeline Connect source control; grant pipeline permissions explicitly; test the pause path before you need it 6 Approval gate on every action Review mode shows Approve/Deny only for Azure infrastructure operations. Emails, Teams posts, and external queries proceed on the agent's reasoning Use hooks or tool access policies to gate non-Azure actions 7 Agent resolves incidents Supported — but during a pilot you don't want it Require human confirmation before resolve. Encode it in the response plan and enforce it with a Stop hook 8 One agent handles everything Response plans route to custom agents; skills cap at five concurrent active Build domain specialists (§4.6). A single mega-agent thrashes its skill budget 9 Autonomous mode by default is fine Per-plan default is Autonomous, and connecting a platform auto-creates an autonomous quickstart_handler Delete the quickstart plan. Set every plan to Review explicitly 10 Agent has broad subscription rights You can't remove individual permissions — only whole resource groups Design resource groups as blast-radius boundaries before onboarding None of these are blockers. All of them are a week of work you'd rather discover now than during your pilot readout. 9. Approval and autonomy policy The policy I would actually ship: Action category Recommended initial policy Read metrics, logs, traces, resource health Automatic Correlate deployments and configuration changes Automatic Create a ServiceNow incident Automatic after deduplication Add sanitized work notes Automatic Prepare a remediation plan Automatic Create a draft ServiceNow change Automatic, but not approve it Modify an Azure production resource Human approval required Execute a VM guest runbook Human approval required Roll back an application Human approval required Delete or replace a stateless VMSS instance Human approval required Resolve a ServiceNow incident Human confirmation during the pilot Delete data, purge queues, replay DLQ messages Separate explicit approval Autonomous remediation Only after a proven, bounded pilot Two notes on making this real: Start in Review and stay there longer than feels necessary. The documented recommendation is to observe for two to four weeks and then promote specific triggers you consistently approve. Not the agent — the triggers. Promotion should be per-response-plan and evidence-based: "we approved this exact rollback proposal eleven times without modification" is a reason to go autonomous. "It seems good" is not. Autonomy should be earned per action type, not per environment. "Autonomous in staging" is a fine starting rule, but the durable version is "autonomous for VMSS single-instance reimage where the workload is stateless and healthy capacity exceeds 80%" — a narrow, well-characterized action with a mechanical precondition. 10. Cross-cutting security controls Use a dedicated managed identity for SRE Agent. Scope Azure roles to selected resources or resource groups. Avoid broad Contributor and Owner assignments. Use fixed-purpose Automation runbooks for guest operations. Do not provide unrestricted SSH, shell, or PowerShell execution. Use a dedicated least-privileged ServiceNow integration identity. Prefer OAuth or a managed connector over stored credentials. Store required secrets in Key Vault — never in prompts. (The agent blocks az keyvault commands entirely, which helps.) Sanitize logs before posting them into ServiceNow. Do not post tokens, personal data, SQL text, message payloads, or memory dumps. Record every approval and production action. Define remediation cooldowns and maximum retry counts. Require post-action application validation. Retain existing manual runbooks as fallback. Use Azure Cost Management alerts for temporary scaling actions. What the platform gives you for free Worth knowing so you don't rebuild it: Layer Isolation model Compute Dedicated sandbox (micro VM) per agent; tool execution separate from the reasoning loop Database Separate database per agent Blob storage Separate blob storage per agent Network Per-agent proxy instance validating every outbound request Credentials Identity sidecar issues short-lived, per-call tokens; credentials never enter the reasoning context Token lifetimes: managed identity ~1 hour (auto-refreshed), OAuth refreshed 20 minutes before expiry, per-tool-call action tokens are single-use, blob SAS 1 hour refreshed at 45 minutes. Each tool invocation launches a fresh process whose entire tree terminates on completion — there are no persistent process pools, so one tool call cannot see another's environment. The audit trail Every az command is logged to your Application Insights as an AgentAzCliExecution custom event. This is your evidence for change management: customEvents | where name == "AgentAzCliExecution" | where timestamp > ago(30d) | project timestamp, command = tostring(customDimensions.command), resource = tostring(customDimensions.resourceId), succeeded = tostring(customDimensions.success), thread = tostring(customDimensions.threadId) | order by timestamp desc Run that query in front of your auditor once and most of the "but can we prove what it did" conversation ends. 11. A 30/60/90 pilot that survives contact with your CAB Phase Enabled capability 1 Detect Azure Monitor alerts 2 Create or correlate ServiceNow incidents 3 Perform read-only investigation 4 Add sanitized findings to ServiceNow 5 Recommend remediation without execution 6 Execute bounded actions after approval in Review mode 7 Validate technical and business recovery 8 Prepare incident resolution and follow-up records 9 Consider autonomy only for proven low-risk actions Phases 1–5 have zero production write risk and deliver most of the MTTR reduction. Do not rush past them to get to the demo-friendly part. The best first five candidates App Service deployment-slot rollback — clean trigger, reversible action, unambiguous validation AKS deployment rollback — same shape, one GitOps caveat VMSS unhealthy-instance replacement — stateless, bounded, easy precondition check Restricted VM disk-recovery runbook — high toil, high value, forces you to build the runbook pattern properly Automatic ServiceNow incident creation and timeline updates — the compounding one; every incident from here on is better documented than any incident before it These five share the properties you want: clear triggers, tightly bounded actions, measurable validation criteria, and practical escalation paths. What to measure in week one Before you enable a single write action, capture your baseline: Median time from alert to first accurate human diagnosis Percentage of incidents where the first hypothesis was wrong Median time from diagnosis to mitigation Percentage of incidents with a complete timeline in the ticket Percentage of incidents that produced a follow-up problem record The read-only phase moves the first, second, and fourth of those immediately. If it doesn't, your telemetry is the problem, not the agent — and that's a genuinely useful thing to discover in week one rather than week twelve. 12. Measuring whether it's actually working Under Monitor → Incident metrics: Metric What it shows Incidents reviewed Total incidents the agent processes Mitigated by agent Resolved autonomously Assisted by agent Agent helped; a human completed it Mitigated by user Human resolved using agent-provided information Pending user action Waiting on a human The counter-intuitive read: "Assisted by agent" and "Mitigated by user" are the healthy numbers during a pilot. A high "Mitigated by agent" count in month one means someone left autonomy on. Watch Pending user action closely. A growing queue there means either your approval routing is broken or the agent is proposing things nobody is comfortable approving — both are important signals, and both are invisible without this dashboard. Also check Monitor → Session insights periodically. Each insight card links back to the thread that generated it, so you can trace any learned pattern to its origin. If the agent has learned something wrong, this is where you find it — and #forget is how you fix it. 13. Resources Core documentation Overview of Azure SRE Agent Security and trust model Agent permissions Run modes Incident response Incident management platforms Incident response plans ServiceNow incident indexing Root cause analysis Execute mitigations Extensibility Custom agents Skills Agent hooks Scheduled tasks Memory and knowledge Closing The framing that makes this work isn't "AI runs my production." It's this: Your agent is the most junior person on the rotation — and the most thorough. It will never skip the Resource Health check. It will never forget to compare against the previous slot. It will never write "restarted it, seems fine" in a work note at 4 AM. And it will never, ever be allowed to rm -rf anything. Every guardrail in this post exists to keep it in that role. Scope the identity to resource groups. Keep production writes in Review. Wrap guest-OS work in runbooks with allowlists. Never let it purge a queue or replay a dead-letter message on its own. Keep the incident open until customers are actually served, not until the graphs look nice. Do that, and the ten workflows above stop being a slide deck and start being your Tuesday. Start with use case #1. One App Service, one slot, one alert rule, one response plan in Review mode. Watch it assemble an evidence chain you'd have spent twenty minutes building by hand, and then decide how much further you want to go. The ten scenarios in this post are target response patterns. Each one requires appropriate telemetry, scoped Azure RBAC, response-plan instructions, approved remediation tooling, and ITSM integration. Confirm current Azure SRE Agent and ServiceNow connector capabilities against Microsoft documentation before implementing — the product is moving quickly, and several capabilities referenced here are in preview.Azure OpenAI Architecture: The Decisions That Actually Matter (Part 3)
Introduction Part 1 of this series tackled the architectural decisions that shape any Azure OpenAI / Microsoft Foundry Models workload — capacity model, deployment scope, governance layer, grounding strategy, and quota engineering. Part 2 turned those decisions into a Well-Architected Framework discipline. Part 3 looks at the part that makes GenAI architecture genuinely different from a traditional service: the platform itself never stops moving. Models are released, promoted to GA, moved to Legacy, deprecated, and eventually retired. New regions come online; certain features (such as Priority Processing) light up only on specific model versions and deployment scopes. Fine-tuned models inherit the lifecycle of their base. Performance characteristics shift between releases. Reliability in this world is not just uptime — it is the ability to absorb continuous change without disrupting production. That discipline is GenAIOps: the people, processes, and tooling that turn model upgrades from emergency events into routine operations. Part 2 already covers the core lifecycle mechanics and upgrade policy trade-offs through a Well-Architected lens. Part 3 stays focused on the operational and architectural practices that make change safe: evaluation of pipelines, observability, routing patterns, prompt governance, and abstraction. Where details are time-sensitive — stage thresholds, SLA windows, regional rollout delays, capacity tier eligibility — they are flagged with "At the time of writing". Always confirm current behavior against Microsoft Learn before committing to a design. Who is this series for? Cloud and Solution Architects Platform and product owners Senior developers responsible for operating Azure OpenAI workloads in production What you’ll learn in Part 3: How to build an evaluation pipeline that promotes model upgrades the way CI/CD promotes code. How to instrument full-stack observability so regressions surface early (latency, errors, token trends, quality drift). How the Model Router pattern, canary releases, and tier-aware fallbacks turn model change into a configuration concern. How to govern prompts as production artifacts with versioning, feature-flagged rollouts, and regression testing. How to manage lifecycle-dependent assets (fine-tuned models) and regional rollout realities without firefighting — plus a GenAIOps Decision Matrix you can reuse as a checklist. We’ve also included a summary decision matrix at the end of this post for quick reference. 1. Model lifecycle (recap) Azure OpenAI/Microsoft Foundry models are living dependencies: new versions are released, promoted from Preview to GA, then eventually move through deprecation toward retirement. To avoid surprises, treat every deployed model version as having an expiration date and design so you can swap versions without rewriting application code. In general, use the Standard deployment auto-upgrade mode that preserves stability but guarantees continuity at retirement, and plan to deliberate blue/green migrations for dedicated (provisioned) capacity where auto-upgrade is not available. For the deeper mechanics (upgrade modes, retirement behavior, and migration playbooks), refer to Part 2’s Reliability section; the rest of this article focuses on the GenAIOps practices that make those upgrades routine. Figure 1 — Models lifecycle 2. GenAIOps: Evaluating Before Promoting Upgrading a model should not be a manual, subjective exercise. Azure AI Foundry provides evaluation capabilities that, combined with a regression prompt suite, turn model upgrades into measurable, repeatable decisions: Side-by-side prompt comparisons across model versions. Automated quality scoring (relevance, coherence, groundedness, safety, and fluency). Structured-output validation (JSON conformance, schema validation). Batch testing across comprehensive prompt libraries representative of real production traffic. Custom evaluation metrics tailored to your domain. Architectural best practice: Maintain a curated regression prompt suite that mirrors real production traffic — including the long tail. Run evaluation pipelines against candidate models before any production cut-over. Integrate evaluation into CI/CD using Azure DevOps, GitHub Actions, or similar automation. Define quality gates that must pass before promotion (e.g., groundedness ≥ a target threshold, p95 latency under a target budget). Pick numbers that fit your workload, not the article. Model promotion should require passing the evaluation gates the same way application code requires passing unit tests. Without automated evaluation, model upgrades become high-risk, low-visibility events that teams avoid until forced by retirement deadlines — the exact pattern that keeps lifecycle work in the "emergency" column instead of the "scheduled" column. Example evaluation workflow: Trigger — a new model version reaches GA, or your migration playbook hits the R-90 step. Deploy — the candidate model goes to a staging deployment. Regress — the prompt suite (typically several hundred to several thousand prompts) is run against the candidate. Compare — the candidate's outputs are scored against the current production model. Inspect — humans review flagged differences; metrics, latency distributions, and cost-per-request go on the dashboard. Gate — an approval step (manual or automated) decides whether the candidate proceeds to blue/green production deployment. Figure 2 — Evaluation Pipeline 3. Observability: Full-Stack or It Didn't Happen GenAIOps is more than one-time evaluation. Once a candidate's model has been promoted, you need continuous, end-to-end observability across the request path — not just at the model boundary. Without it, you are operating blind during model transitions. At a minimum, instrument: Prompt processing time (gateway through model invocation). Model inference latency, expressed as p50, p95, and p99 — averages hide the experience of the slowest 5% of users. Token consumption (prompt tokens, completion tokens, total) trended over time. Error rates by class (429 throttling, 503 service unavailable, 400 validation errors, content-filter rejections). Model version distribution — which versions are actually serving traffic right now. User-satisfaction signals (thumbs-up/down, explicit feedback, session abandonment). Many performance regressions only surface at scale. A model version that performs well in evaluation against a few hundred prompts may behave differently under production traffic patterns. Plan for that. A practical metrics architecture on Azure tends to combine: Application Insights for end-to-end request tracing across the application and gateway. Azure Monitor for infrastructure, quota, and PTU utilization of metrics. Custom telemetry for prompt-level success/failure tracking and quality scoring. Log Analytics for forensic analysis when a regression is suspected. Drift in model behavior rarely shows up as a single broken request — it surfaces as a slow shift in tail latency, fallback rate, or user-satisfaction signal. Monitoring that only looks at average will miss it. 4. The Model Router Pattern As GenAI systems mature, a static single-model architecture becomes both limiting and expensive. A Model Router introduces dynamic, intelligent model selection in front of one or more model deployments. Typical responsibilities of a router: Send simple queries to a smaller, faster model and complex reasoning to a larger one. Run canary releases of new model versions with percentage-based rollouts. A/B test model variants to measure quality, latency, and cost differences. Route to the right capacity tier — including falling back from Provisioned to Standard during migrations or capacity constraints. Where the workload also needs lower-variance latency on the Standard side, route latency-critical traffic through Priority Processing on a Global Standard or Data Zone Standard (US) deployment, on a model version that supports it. (At the time of writing, Priority Processing is enabled by setting the service_tier attribute on the request and requires a model version released on or after 2025-12-01 — verify both eligibility constraints on Microsoft Learn before depending on it. Decision logic can be driven by any combination of: Query complexity (simple heuristics or a lightweight classifier). User tier (e.g., free vs premium). Response-time requirements (interactive vs background). Cost constraints — pick the cheapest model that meets the quality bar. Regional model availability and capacity. Implementation options: Azure API Management — built-in routing policies, weighted backends, retry policies. Azure Front Door — global routing with health probes. Custom routing service — maximum flexibility, more operational overhead. Semantic Kernel or LangChain — framework-level routing logic embedded in the application. Beyond cost and performance, the Model Router pattern decouples the application layer from any single model version. That decoupling is what makes lifecycle management tractable: when a model moves to Legacy, you change a router rule, not application code. Figure 3 — Model Router Architecture vs Blue/Green Deployment 5. Prompt Lifecycle Governance Prompts are not strings embedded in code. They are production artifacts that influence quality, cost, and safety, and they evolve almost as often as the models behind them. Treat them as first-class assets. Prompt templates Separate stable system instructions from dynamic content (user input and retrieved context). This lets you version, test, and audit each layer independently. Version control Store prompts in Git — full history, code review, branching, and tagging. Treat prompt changes the way you treat code changes: pull request, review, and test before merging. Feature-flagged rollouts Roll out prompt changes gradually using feature flags. Monitor the impact on a subset of users before exposing the change broadly. The same observability stack that watches model upgrades should watch prompt rollouts. Regression testing Maintain a regression suite of expected prompt behaviors and run it whenever prompts or models change. The suite reuses the same evaluation pipeline you built in Section 2. Prompt-level metrics Success rate — did the prompt achieve its intended outcome? Fallback rate — how often did users rephrase or abandon? Satisfaction score — explicit user feedback. Token efficiency — average tokens per successful completion (a leading indicator of cost regression). PII and privacy safeguards Customer prompts and completions are not used to train base models. That means logging is safe for debugging — but defense in depth still applies: Redact PII (names, emails, phone numbers, addresses) before logs are written. Apply RBAC on Log Analytics workspaces so only the right roles can access raw prompt data. Govern data retention with automated purging after a defined window. Keep audit trails of who accessed which logs and when. Prompt quality is not a one-time effort. It is an ongoing operational discipline that needs tooling, processing, and measurement, in the same way application code does. Figure 4 — Prompt Lifecycle Governance 6. Fine-Tuned Models: The Hidden Retirement Risk Fine-tuned models inherit the lifecycle of their base model. That creates a cascading retirement risk that many teams overlook. During base-model deprecation: New fine-tuning jobs against that base are blocked — you can no longer create new fine-tuned versions. Existing fine-tuned deployments continue serving inference, with no immediate impact. When the base model is retired: Fine-tuned deployments stop responding (HTTP 404), exactly like any other deployment pinned to a retired version. The migration imperative is straightforward: retrain fine-tuned models on the successor base model well before the retirement date, ideally during the predecessor's Legacy or Deprecated stage. Architectural considerations: Track base-model dependencies explicitly in your asset inventory — the same place you track library and runtime versions. Schedule retraining workflows aligned with base-model lifecycle dates, not with team availability. Validate fine-tuned model quality on the new base; behavior can shift between base versions. Keep training datasets in version-controlled storage, so retraining is reproducible. Re-evaluate whether fine-tuning is still necessary; newer base models, combined with better prompting (few-shot, chain-of-thought, structured outputs), sometimes remove the need entirely. Common mistake: investing heavily in fine-tuning without budgeting for the recurring retraining cost and lifecycle overhead. Improved prompting on a newer base model is often the cheaper path. 7. Regional Rollouts and Multi-Region Strategy Successor models are not always available in every Azure region simultaneously. Microsoft typically releases a new version in a subset of regions first, with broader rollout following over weeks or months. At the time of writing, the regional rollout schedule is published per model on Microsoft Learn — confirm before assuming a particular region will receive a release on a particular day. Maintain staging deployments in early-release regions Even if production runs elsewhere, maintain a staging deployment in regions that tend to receive new models earliest. That gives you visibility into the successor's behavior before it auto-upgrades into your primary region. Pre-test successor models before primary auto-upgrades If your production deployment uses "Once the current version expires", the upgrade will happen automatically. Pre-testing in an early-release region lets you catch behavioral changes before they hit live traffic. Multi-region routing for lifecycle flexibility Azure Front Door or Azure API Management with multi-region back-ends lets you route based on model availability, capacity headroom (one region may have quota while another is exhausted), and latency. Combined with the Model Router pattern from Section 6, this turns regional staggering from a constraint into an option. Account for capacity-tier eligibility in your routing Some capacity tiers are scoped to specific deployment scopes — Priority Processing, for example, is offered on Global Standard and Data Zone Standard (US) deployments at the time of writing. Bake those eligibility constraints into routing rules, so a fallback path does not silently land in an ineligible deployment. Multi-region strategy is no longer just a disaster-recovery concern. It is also lifecycle resilience — the ability to test, stage, and absorb model changes without coupling your platform to a single region release schedule. 8. Future-Proofing Through Abstraction Future-proofing is architectural, not procedural. The goal is to design systems that adapt to change without requiring code rewrites every time a model is promoted, deprecated, or retired. Abstract model calls behind a service layer Avoid calling Azure OpenAI APIs directly from the application code. Introduce an internal Model Service that owns model selection, retry and fallback, prompt-template lookup, and response validation. The application asks for an outcome ("summarize this", "classify that"); the Model Service decides which model and which prompt to use. Externalize model names and configuration Store model identifiers, versions, and parameters in configuration or feature flags — never as hard-coded strings. Changing models then becomes a configuration change, not a deployment. Centralize prompt logic Maintain prompts in a registry or template repository, not scattered across codebases. This enables centralized versioning, A/B testing without code changes, and prompt optimization that is decoupled from application releases. Avoid scattering model identifiers across the codebase Use constants, enums, or configuration references rather than literal model strings repeated across many files. The number of files that have to change at upgrade time is a leading indicator of how painful the upgrade will be. Benefits of abstraction: Seamless model replacement — swap models without touching application logic. Multi-model strategies — the Model Router pattern becomes trivial to add. Provider flexibility — integrating additional or alternative providers becomes a service-layer change, not an application to rewrite. Faster adoption of new capabilities — reasoning controls, function calling, structured outputs land in one place. Common mistake: Prototyping with direct API calls for speed and never refactoring. The technical debt accumulates until a model upgrade requires an emergency engineering sprint. Figure 5 — Abstraction Layer for Future-Proofing. Final Perspective The most important shift this article asks for is a change in operational mindset: Model upgrades are not emergencies. They are scheduled events. Retirement deadlines are not surprising. They are published timelines, often with months of notice. Architecture fails when teams treat models as static dependencies. They succeed when they treat models as evolving infrastructure. In practice, GenAIOps means: Automated evaluation that runs continuously, not just during migrations. Controlled rollouts using blue/green or canary patterns. Observability-driven decisions based on metrics, not intuition. Lifecycle-aware planning, with retirement dates tracked alongside library and runtime upgrades. Modular design that decouples applications from specific model versions. Across the three parts of this series we have covered the architectural decisions that frame an Azure OpenAI / Microsoft Foundry Models workload (Part 1), the Well-Architected Framework discipline that keeps it sustainable (Part 2), and the GenAIOps practices that let it evolve without firefighting (Part 3). The organizations that succeed long-term are the ones that plan for model evolution from day one, invest in evaluation and observability tooling, decouple application logic from model specifics, and treat prompts and configurations as versioned artifacts. Generative AI architecture is not about deploying a model endpoint. It is about building a platform that absorbs change gracefully as the AI landscape shifts. The retirement of a model should be a routine operational event, not a crisis. If your architecture makes model upgrades feel risky or expensive, refactor before the next retirement deadline forces your hand. Lifecycle & GenAIOps Decision Matrix Use this as a checklist when reviewing or signing off on the GenAIOps posture of an Azure OpenAI / Microsoft Foundry Models platform. One row per decision; one rule of thumb per row. Area Decision Rule of thumb Watch out for Lifecycle Version expiry tracking Treat model versions as expiring dependencies: inventory every deployed model/version, track deprecation/retirement dates, and design so swapping versions is a configuration change (details on upgrade modes in Part 2). Pinning versions without an owner; discovering retirement dates after an outage or emergency migration window. Evaluation Promotion gates Pass the regression suite + meet domain-specific quality and latency thresholds before promoting any model. Subjective "feels better" sign-off; gates that exist on paper but never block a release. Evaluation Pipeline integration Evaluation runs in CI/CD on every candidate; the same suite watches prompt changes. Manual evaluation runs that only happen under retirement pressure. Observability Latency and error metrics Track p50/p95/p99 latency, 429/503/4xx rates, token trend, and model-version distribution. Alert on tail latency and sustained throttling. Average-only dashboards; missed Service Health notifications for model retirements. Observability Quality drift Trend per-prompt success rate, fallback rate, and user-satisfaction signals; surface drift before users complain. Treating quality as a one-time evaluation event. Architecture Model Router Centralize model selection, canary, and fallback (including Priority Processing on eligible deployments) behind a router service. Application code that calls a specific model deployment by name; routing logic scattered across services. Architecture Abstraction layer Application code asks for an outcome; the Model Service decides which model and prompt; configuration drives model selection. Hard-coded model identifiers across many files; bypass paths that skip the service layer. Prompts Prompt governance Prompts in Git, behind feature flags, with regression tests, prompt-level metrics, and PII redaction in logs. Prompts copy-pasted across services; PII in logs; no rollback path for a regressed prompt. Fine-tune Fine-tuned model lifecycle Track fine-tuned models against base-model dates; schedule retraining during the predecessor's Legacy/Deprecated window. Treating fine-tuned models as permanent infrastructure; lost or unversioned training datasets. Regional Multi-region for lifecycle resilience Maintain staging in early-release regions; route across regions to absorb staggered rollouts and capacity gaps. Single-region production with no early-release staging; routing rules that ignore tier-eligibility constraints. Disclaimer I am a Microsoft employee. The views and opinions expressed in this article are my own and do not necessarily reflect those of Microsoft. This content is informational and educational; it is not an official Microsoft statement, recommendation, or commitment. Service tiers, model availability, lifecycle stages, deprecation timelines, regional rollouts, pricing, and SLAs evolve — always validate against the latest Microsoft Learn documentation before making architectural or migration decisions. References Azure OpenAI model deprecations and retirements Working with Azure OpenAI models — versioning and upgrades Provisioned throughput for Azure OpenAI Enable Priority Processing for Microsoft Foundry Models Azure AI Foundry — evaluation of generative AI applications Monitor Azure OpenAI Azure API Management — GenAI Gateway capabilities Azure Front Door routing for AI back-ends Use managed identities with Azure OpenAI Azure AI Content Safety Fine-tune models with Azure OpenAI Azure Well-Architected FrameworkDatabricks Lakebase: The operational database for AI agents and apps
Understanding the Evolution: From Lakehouse to Lakebase The modern data landscape has long been characterized by a fundamental schism: Online Transaction Processing (OLTP) systems, designed for high-frequency, low-latency transactions in applications, and Online Analytical Processing (OLAP) systems, optimized for complex queries, reporting, and machine learning on vast datasets. This division historically necessitated intricate and often fragile Extract, Transform, Load (ETL) processes to move and synchronize data between these disparate environments, leading to increased complexity, data duplication, and governance challenges. Databricks Lakehouse architecture emerged to unify data warehousing and data lake f unctionalities for analytical workloads, offering the flexibility of data lakes with the performance and governance of data warehouses. However, a critical piece remained: native, high-performance OLTP capabilities directly within this unified environment. This is where Databricks Lakebase enters the picture, representing a significant evolution by bringing fully managed PostgreSQL OLTP capabilities directly into the Databricks Data Intelligence Platform. Lakebase addresses the need for a single, governed platform that can seamlessly handle both transactional and analytical workloads, thereby simplifying data architectures, reducing operational overhead, and accelerating the development of real-time applications and AI agents. By integrating OLTP at the core of the lakehouse, Databricks aims to create a truly unified data and AI platform. The Architectural Innovation: Separation of Compute and Storage At the heart of Databricks Lakebase's efficiency and scalability lies its innovative architecture, which fundamentally separates compute from storage. Unlike traditional monolithic databases where these components are tightly coupled, Lakebase decouples them, offering distinct advantages: Elastic Scaling and Cost Efficiency The transactional compute layer in Lakebase is serverless and ephemeral, meaning it can scale up or down dynamically based on demand. This includes the ability to scale to zero during periods of inactivity, significantly optimizing cost by ensuring you only pay for the compute resources actively used. Data, on the other hand, is persisted directly into low-cost, durable cloud object storage (e.g., Azure Blob Storage) using open formats like Delta Lake. This design not only reduces storage costs but also prevents vendor lock-in and allows other engines within the Databricks platform to access the data directly. Open Data Formats and Interoperability By storing data in open formats, Lakebase ensures high interoperability within the Databricks ecosystem and beyond. This approach eliminates the need for complex and time-consuming ETL processes to move transactional data to the analytical layer, as the data is inherently accessible to both. This foundational integration streamlines data pipelines and provides a unified view of data across all workloads. Key Technical Capabilities and Features Databricks Lakebase offers a rich set of features that make it a compelling solution for modern data architectures: PostgreSQL Compatibility: Lakebase provides full PostgreSQL semantics, including ACID transactions, indexing capabilities, and support for standard JDBC/psql clients. This familiarity allows developers to leverage existing skills and tools, minimizing the learning curve. Fully Managed Service: Databricks handles the complexities of provisioning, scaling, patching, backups, and ensuring high availability, freeing up development teams to focus on application logic rather than database administration. Managed Change Data Capture (CDC): A crucial feature, managed CDC ensures that operational data in Lakebase remains synchronized with Delta Lake tables for analytical consumption. This continuous synchronization is vital for keeping BI models and AI applications updated with the freshest transactional data. Autoscaling (Lakebase Autoscaling): The latest iteration of Lakebase features intelligent autoscaling of compute resources. It dynamically adjusts Compute Units (CU) based on various metrics like CPU load, memory usage, and working set size, preventing performance bottlenecks and out-of-memory (OOM) issues. It also supports branching and instant restore, enhancing developer agility and operational resilience. Databricks Apps Synergy: Lakebase is designed to serve as the transactional backend for Databricks Apps, enabling the creation and deployment of interactive applications directly on the platform, leveraging governed data and powerful analytics. Governance, Security, and Cost Efficiency with Lakebase Adopting Databricks Lakebase brings significant benefits in terms of data governance, security, and overall cost management, aligning with the principles of a modern data intelligence platform. Unified Governance through Unity Catalog One of Lakebase's most powerful integrations is with Unity Catalog, Databricks' unified governance solution. This integration provides a single pane of glass for managing data assets across the entire Databricks Data Intelligence Platform. Lakebase databases can be registered as catalogs within Unity Catalog, extending its robust governance framework to operational data. This means: Consistent Access Control: Policies defined for your lakehouse data automatically apply to Lakebase, ensuring uniform security and access management across both operational and analytical workloads. Centralized Auditing and Lineage: Unity Catalog provides comprehensive auditing capabilities and data lineage tracking for Lakebase assets, simplifying compliance and offering transparent insights into data flows. Simplified Security Management: By unifying governance, organizations can reduce the complexity of managing security policies across disparate systems, enhancing overall data security posture. Robust Security and Data Protection Lakebase is designed with enterprise-grade security in mind, leveraging existing cloud infrastructure and Databricks' security features: Network Integration: It integrates seamlessly with cloud networking services (e.g., Azure Private Link) for secure, private connectivity. Identity Management: Integration with enterprise identity providers (e.g., Microsoft Entra ID) ensures secure authentication and authorization. Data Encryption: Data is encrypted at rest and in transit, protecting sensitive information throughout its lifecycle. High Availability and Disaster Recovery: As a fully managed service, Lakebase inherently provides features for high availability and point-in-time recovery, ensuring operational resilience. Optimized Cost Efficiency The architectural separation of compute and storage, coupled with advanced autoscaling capabilities, contributes to significant cost savings compared to traditional database architectures: Pay-as-you-go Compute: With serverless and autoscaling compute, you only pay for the resources consumed during active processing, with the ability to scale down to zero when idle. Low-Cost Storage: Leveraging economical cloud object storage for data persistence drastically reduces storage costs. Reduced ETL Overhead: By eliminating the need for complex ETL pipelines between OLTP and OLAP, organizations save on infrastructure, development, and maintenance costs associated with data movement and transformation. This can lead to reported savings of 40-50% in many environments. Lakebase in Action: Powering Real-Time Applications and AI Agents Databricks Lakebase opens up new possibilities for building intelligent, data-driven applications that require both transactional capabilities and deep analytical insights. Its unified approach simplifies development and accelerates time-to-market for innovative solutions. Real-World Use Cases Personalized Recommendations: Build real-time recommendation engines that leverage fresh transactional data from Lakebase to provide immediate and highly relevant suggestions to users. Customer Segmentation and Real-Time Updates: Maintain and update customer profiles and segments in real-time, enabling personalized experiences and targeted marketing campaigns. Feature Stores for Machine Learning: Utilize Lakebase as a feature store to serve low-latency features to AI models, ensuring that predictions and decisions are based on the most current data. Stateful AI Agents: Develop AI agents that can maintain conversational state and interact dynamically with users, using Lakebase as a reliable backend for transactional data. Order Processing Systems: Implement operational applications that require high-frequency reads, writes, and updates, such as order management or inventory systems, directly on the Databricks platform. Interactive Workflow Tools: Create interactive data applications and dashboards that allow users to both view analytical insights and perform transactional updates within the same environment. A Practical Code Snippet Developing with Lakebase feels familiar due to its PostgreSQL compatibility. Here’s a simple example demonstrating basic CRUD (Create, Read, Update, Delete) operations within a Lakebase table: -- Create a schema for your application CREATE SCHEMA app AUTHORIZATION CURRENT_USER; -- Create a table to store session data for an AI agent CREATE TABLE app.sessions ( session_id UUID PRIMARY KEY, user_id TEXT NOT NULL, state JSONB NOT NULL, created_at TIMESTAMPTZ DEFAULT now(), updated_at TIMESTAMPTZ ); -- Create an index to optimize queries on agent status CREATE INDEX ON app.sessions ((state->>'agentStatus')); -- Insert a new session record INSERT INTO app.sessions(session_id, user_id, state) VALUES (gen_random_uuid(), 'u-123', '{"agentStatus":"active","score":0.82}'); -- Update an existing session's state UPDATE app.sessions SET state = jsonb_set(state, '{score}', '0.91'::jsonb), updated_at = now() WHERE user_id='u-123'; -- Query active sessions SELECT user_id, state->>'score' as current_score FROM app.sessions WHERE (state->>'agentStatus') = 'active'; This SQL snippet showcases how developers can interact with Lakebase using standard PostgreSQL syntax, enabling rapid application development within the Databricks environment. The Lakebase Advantage: Performance and Reliability Beyond its unified architecture, Lakebase is engineered for predictable performance and robust reliability, essential for mission-critical operational applications. The radar chart above provides an opinionated comparison of Databricks Lakebase against traditional OLTP systems across several key attributes. Lakebase demonstrates superior performance predictability, dynamic scalability, cost efficiency, and ease of management, coupled with strong data governance due to its integration with Unity Catalog. Traditional OLTP systems, while effective for their specific purposes, often score lower in these cloud-native, unified data platform metrics. Reliability Features for Business Continuity Lakebase integrates several critical reliability features that ensure business continuity and data integrity: Branching: This feature allows developers to create isolated, production-like environments for testing changes without affecting the main operational database. It promotes safer development practices and faster iteration cycles. Instant Restore and Point-in-Time Recovery (PITR): In the event of data corruption or accidental deletion, Lakebase enables quick restoration to a previous state, minimizing downtime and ensuring data resilience. High Availability: As a managed service, Lakebase is designed for high availability, with automated failover mechanisms and robust infrastructure ensuring continuous operation. Validation and Troubleshooting: Ensuring a Smooth Lakebase Experience Successful implementation and ongoing operation of Databricks Lakebase rely on proper validation and an understanding of common troubleshooting steps. This section provides a framework for ensuring your Lakebase deployment meets performance and reliability expectations. An introductory video to Lakebase, explaining its core functionality and benefits for data apps and AI agents. Key Validation Steps After provisioning and configuring your Lakebase instance, it's crucial to perform a series of validation tests: Connectivity Verification: Confirm successful connections from your applications or development tools (e.g., psql, JDBC clients) to the Lakebase instance. Ensure that Unity Catalog registration is visible and properly configured for governance. Performance Baseline: Conduct baseline QPS (Queries Per Second) tests and monitor latency under expected load conditions. Validate that autoscaling events occur as anticipated and that performance targets are met. Data Synchronization (CDC): Test the end-to-end data flow by inserting/updating records in Lakebase and verifying their timely appearance in Delta Lake tables via managed CDC. If reverse synchronization (Delta to Lakebase) is configured, validate that as well. Governance and Security Checks: Confirm that Unity Catalog permissions are correctly enforced for Lakebase assets and that audit logs accurately reflect data access and modification events. Verify network security configurations (e.g., Private Link) are functioning as intended. Common Troubleshooting Scenarios While Lakebase is designed for stability, understanding potential issues and their resolutions is key to efficient operation: Problem Area Symptom Potential Cause(s) Troubleshooting Step(s) Performance High latency, slow queries, autoscaling not triggering as expected. Inefficient queries, missing indexes, insufficient compute resources, working set exceeding memory. Inspect query plans, add appropriate indexes, monitor CU utilization, review autoscaling logs, consider increasing initial compute capacity if persistently underperforming. Data Sync (CDC) Stale data in Delta Lake, sync job failures, data inconsistencies. Incorrect Unity Catalog permissions, CDC configuration errors, network issues, regional feature limitations. Verify Unity Catalog access for CDC process, check CDC job logs for errors, confirm network connectivity between Lakebase and Delta Lake, consult Databricks documentation for regional CDC availability. Connectivity Unable to connect from application, authentication failures. Incorrect connection strings, firewall rules blocking access, misconfigured private endpoints, invalid credentials/tokens. Double-check connection parameters, review network security group (NSG) and firewall rules, validate Private Link configuration, ensure correct user/service principal credentials. Governance Unauthorized access, unexpected data visibility, audit log discrepancies. Incorrect Unity Catalog access policies, schema mismatches, misconfigured external locations. Review and refine Unity Catalog grants on Lakebase catalogs and schemas, verify external location configurations, ensure consistent data object naming conventions. Feature Limitations Specific PostgreSQL features or extensions not working. Managed environment restrictions, unsupported extensions. Consult Databricks documentation for supported PostgreSQL versions and extensions in Lakebase. Adapt application logic to use supported alternatives if necessary. By proactively monitoring and understanding these aspects, Cloud Solution Architects can ensure robust and efficient operation of Lakebase within their Databricks ecosystem. Conclusion Databricks Lakebase represents a pivotal advancement in data architecture, fundamentally reshaping how organizations approach operational and analytical workloads. By seamlessly integrating a fully managed PostgreSQL OLTP engine directly into the Databricks Data Intelligence Platform, Lakebase addresses the long-standing challenge of data fragmentation. This unification not only simplifies complex ETL processes and reduces operational overhead but also extends robust governance and security through Unity Catalog across the entire data estate. The innovative separation of compute and storage, coupled with intelligent autoscaling, delivers unparalleled cost efficiency and dynamic performance. For Cloud Solution Architects, Lakebase offers a compelling path to building scalable, real-time applications and sophisticated AI agents, leveraging fresh transactional data alongside comprehensive analytical insights—all within a single, consistent, and highly performant environment. This strategic evolution of the lakehouse architecture empowers enterprises to unlock new levels of agility, innovation, and data-driven decision-making.Migrating Azure Data Factory and Synapse Pipelines to Fabric Data Factory
Migrating data pipelines from Azure Data Factory (ADF) and Azure Synapse Pipelines to Microsoft Fabric Data Factory represents a significant modernization opportunity and a catalyst for accelerating AI innovation across the enterprise. With Fabric Data Factory, customers can unify their data estate, streamline data engineering workflows, and more effectively leverage real-time analytics, generative AI, and machine learning at scale. This article outlines the key technical considerations for a successful migration from ADF/Synapse pipelines to Fabric Data Factory. Fabric Data Factory vs. ADF and Synapse Pipelines: What’s Different? Fabric Data Factory is officially described by Microsoft as the next generation of Azure Data Factory, built to handle your most complex data integration challenges with a simpler, more powerful approach. It retains ADF’s core engine capabilities while introducing major improvements enabled by Fabric’s unified, AI-centric platform including OneLake, expanded activities and native Copilot experiences. A fundamental shift is the move to a fully managed SaaS model, with several important differences: No infrastructure management: Fabric eliminates Azure Integration Runtimes entirely. Compute is managed automatically within a Fabric capacity. For on‑premises connectivity, the On‑Premises Data Gateway (OPDG) replaces ADF’s Self‑Hosted Integration Runtime. No publish step: Pipelines are authored directly in the Fabric portal and can be saved or executed immediately, removing the separate publish step required in ADF. Simplified data connections: Traditional Linked Services and Datasets are replaced by Connections and inline data properties within activities, reducing configuration complexity. New native activities: Fabric introduces capabilities not available in ADF/Synapse pipelines, including Office 365 Outlook email, Teams messaging, semantic model refresh, Fabric notebooks, Invoke SSIS (preview), and Lakehouse maintenance (preview). Enhanced CI/CD: Built‑in deployment pipelines support cherry‑picking, individual item promotion, Git integration, and SaaS‑native CI/CD beyond ADF’s ARM template–based approach. AI Copilot: Fabric Data Factory includes Copilot to assist with pipeline creation and management, a capability not available in ADF or Synapse pipelines. For more details see: Differences between Data Factory in Fabric and Azure - Microsoft Fabric | Microsoft Learn Common Migration Challenges and Recommended Mitigations Migrating to Fabric Data Factory introduces new choices and challenges. While the move to Fabric offers substantial benefits, success depends on understanding key differences, migration challenges and planning accordingly. The table below summarizes the most important considerations to help guide a smooth and successful transition. Table 1. Migration Challenges and Mitigation Challenge Description Recommended Mitigation Feature Gaps Some ADF/Synapse features (e.g., SSIS IR, Managed VNets, certain triggers) are not yet fully supported in Fabric. Delay migration of affected pipelines or redesign using Fabric‑native alternatives. Monitor updates via the https://roadmap.fabric.microsoft.com Mapping Data Flows ADF Mapping Data Flows don’t directly map to Fabric equivalents. Rebuild using Dataflow Gen2, Fabric Warehouse SQL, or Spark notebooks. Validate transformation logic and data types post‑migration. Trigger Redesign Fabric lacks centralized trigger management; scheduling must be defined at the pipeline level. Recreate triggers per pipeline and apply standardized naming conventions and documentation to maintain operational clarity. Global Parameters ADF Global Parameters must be converted to Fabric Variable Libraries. Use Microsoft’s conversion guidance and account for differences in data types and runtime usage patterns. See Convert Azure Data Factory Global Parameters to Fabric Variable Libraries. Dynamic Connections Fabric does not support dynamic linked service properties in the same way as ADF. Parameterize connection objects within pipeline activities using dynamic content. Deployment Performance Some environments report slower execution of deployment pipelines in Fabric. Break deployments into smaller logical units and validate performance during pilot phases prior to production rollout. Capacity Planning Fabric uses a fixed‑capacity compute model instead of ADF’s elastic pay‑as‑you‑go runtime. Right‑size Fabric capacity based on peak load testing and continuously monitor usage with tools such as the Fabric Capacity Estimator. Migration Tooling Migration Assistant: Microsoft Fabric includes a built‑in Migration Assistant for both ADF and Synapse pipelines, designed specifically to support pipeline migrations. To assess migration readiness, open your ADF/Synapse pipeline instance, go to the authoring canvas, and select Migrate to Fabric (Preview) > Get started (Preview). As shown in the assessment summary below, pipelines are grouped into migration readiness categories such as Ready, Needs Review, Coming Soon, and Unsupported. This classification gives engineering teams early visibility into potential migration risks by highlighting activities or configurations that may behave differently in Fabric and require validation or adjustment after migration (Needs review), features that are not currently supported in Fabric but are planned for future availability (Coming soon), or not available in Fabric and will require redesign or re‑implementation (Unsupported). In enterprise environments with large pipeline estates, this insight is critical for avoiding unexpected failures or delays during migration. After completing the assessment, you can proceed with the migration wizard and mount your ADF pipelines into Microsoft Fabric. Mounting does not migrate your ADF pipelines to Fabric Data Factory at this stage. Instead, it creates a reference to your existing instances within the Fabric workspace without consuming Fabric capacity. After mounting, run pipelines side by side to validate behavior and results. Once the side by side has been validated, select Migrate to Fabric button to proceed with connection mapping and the actual migration to Fabric Data Factory. After completing the migration process, you will be presented with the Migration Results page. This view provides a summary of all selected pipeline resources along with their migration status and corresponding Fabric resource names. Successfully migrated pipelines are now available as Fabric‑native items within the workspace, while any errors or unmapped dependencies are flagged for further review. For Synapse Analytics pipelines, you transition directly into the Fabric Data Factory experience (assess->map->migrate flow) rather than mounting first to reference Synapse pipelines externally. For detailed migration steps, follow this link: Assess your Azure Data Factory and Synapse pipelines for migration to Fabric - Azure Data Factory | Microsoft Learn PowerShell automation tool: Microsoft provides a PowerShell upgrade utility to accelerate migration from Azure Data Factory to Fabric Data Factory. Using the Microsoft.FabricPipelineUpgrade module, you can translate a large subset of ADF pipeline JSON into Fabric‑native definitions, giving you a fast, scalable starting point for migration. The tool covers common patterns such as Copy, Lookup, Stored Procedure, and standard control flow. Manual follow‑up is still required for edge cases (custom connectors, complex expressions, and some data flow scenarios). Import-AdfFactory -SubscriptionId <your Subscription ID> -ResourceGroupName <your Resource Group Name> -FactoryName <your Data Factory Name> -PipelineName "pipeline1" -AdfToken $adfSecureToken | ConvertTo-FabricResources | Export-FabricResources -Region <region> -Workspace <workspaceId> -Token $fabricSecureToken For step‑by‑step guidance, see: Detailed Tutorial for PowerShell-based Migration of Azure Data Factory Pipelines to Fabric - Microsoft Fabric | Microsoft Learn Open‑Source Migration Tooling In addition to Microsoft‑supported migration utilities, the Fabric Toolbox provides a set of open‑source tools designed to assist with migration planning, readiness analysis, and pipeline translation from ADF and Synapse to Fabric Data Factory. Fabric Data Factory Migration Assistant PowerShell: An open‑source tool from the Fabric Toolbox that supports migration from both Azure Data Factory and Synapse ARM templates and built as a browser‑based single‑page application (SPA). https://github.com/microsoft/fabric-toolbox/tree/main/tools/FabricDataFactoryMigrationAssistant Fabric Assessment Tool: An open‑source command‑line utility used to connect to and scan workspaces in order to extract inventory data and assess migration scope by creating a structured export of assets for planning and analysis. https://github.com/microsoft/fabric-toolbox/tree/main/tools/fabric-assessment-tool When to Use What? Organizations typically adopt one of three migration strategies when transitioning ADF or Synapse pipelines to Fabric Data Factory: Lift‑and‑Shift to accelerate transition timelines with minimal pipeline refactoring. Modernization to re‑architect orchestration logic and fully leverage Fabric‑native analytics and AI capabilities. Hybrid to balance migration velocity with targeted modernization of high‑value or low‑parity workloads. The appropriate migration paths should be aligned with business priorities, existing integration patterns, and the desired pace of platform transformation, and is largely determined by the feature parity between existing ADF/Synapse assets and their Fabric Data Factory equivalents. A range of migration tooling options are available depending on migration scope and pipeline complexity: Built-In Fabric UI Assistant – Migrate to Fabric: Use this assistant to assess pipeline readiness across both ADF and Synapse environments, mount existing ADF pipelines into a Fabric workspace, perform side‑by‑side validation, or migrate supported Synapse pipelines directly into Fabric Data Factory experience. PowerShell Upgrade Tool (Microsoft‑supported): Use this for bulk ADF migrations at scale, repeatable upgrades, and CI/CD‑driven pipeline conversion with a supported path. Fabric Data Factory Migration Assistant PowerShell (Open Source): Use for early analysis, connector mapping, and generating a migration starting point outside the Fabric UI. Fabric Assessment Tool (Open Source): Use before migration to understand scope, inventory, dependencies, and readiness across your Fabric and data estate. Manual migration: best suited for complex, low‑parity pipelines and provides an opportunity to modernize architecture using Fabric’s native capabilities, delivering long‑term benefits in maintainability, performance, and cost. Key Considerations for a Smooth Transition Before migrating, it’s important to understand the architectural differences between Azure Data Factory or Synapse pipelines and Fabric Data Factory. Reviewing these differences early helps determine which pipeline components can be reused, translated, or redesigned for Fabric‑native execution. Start by prioritizing low‑risk, high‑parity pipelines that can be migrated with minimal redesign. Mounting existing ADF pipelines into Fabric enables gradual migration and side‑by‑side testing, allowing teams to validate compatibility before using conversion tools or replatforming workloads. For larger environments, the Microsoft.FabricPipelineUpgrade PowerShell module or Open-Source tools can be used to migrate pipelines at scale while mapping linked services to Fabric connections. Where possible, leverage Fabric‑native capabilities such as Copilot for pipeline authoring, and code fix, deployment pipelines for CI/CD, and OneLake shortcuts to access external data without duplication. It’s also recommended to validate migrated pipelines under production‑like workloads to confirm performance and reliability before cutover. For complex or large‑scale enterprise migrations, engaging Microsoft partners can help accelerate modernization efforts while minimizing operational risk. Partners | Microsoft Fabric For detailed best practices guidance, refer to: Migration Best Practices for Azure Data Factory to Fabric Data Factory - Microsoft Fabric | Microsoft Learn Summary Migrating from Azure Data Factory or Synapse pipelines to Microsoft Fabric Data Factory represents a key step toward building a unified, AI‑ready analytics platform. By leveraging the built‑in migration assessment and associated tooling, organizations can perform pipeline‑level compatibility analysis, identify unsupported activities or configuration dependencies, and implement a phased modernization strategy aligned with workload readiness. Successful transitions require a clear understanding of the architectural shift from ADF/Synapse’s PaaS to Fabric’s SaaS‑managed model, where compute is fully managed within the Fabric capacity, traditional Integration Runtimes are no longer required, and datasets and linked services are replaced with connection‑based configurations defined inline within pipeline activities. By adopting Fabric‑native capabilities such as deployment pipelines for CI/CD, Copilot‑assisted pipeline authoring, and OneLake, organizations can standardize pipeline lifecycle management, enable governed access to shared data assets across domains, and support multi‑cloud integration through virtualized data access allowing pipelines to operate on distributed datasets without duplicating or relocating data across Lakehouse, Data Warehouse, and Real‑Time Analytics workloads within a unified Fabric workspace.Create an Organizational Assets Library (including Multi-Geo & Information Barriers guidance)
Overview This guide walks through a practical approach to setting up SharePoint Online (SPO) Organizational Assets Libraries (OAL). It includes optional guidance for more complex tenants—such as Multi-Geo and Information Barriers (IB) - because those scenarios are often under-documented. What you’ll accomplish: Create and register Organizational Assets Libraries so templates, fonts, and brand images are available in Office apps, with notes for Multi-Geo, Information Barriers, Brand Center, and Copilot integration where applicable. Applies to: Standard (single-geo) tenants, Multi-Geo tenants, tenants with Information Barriers, and environments using Brand Center and/or Copilot features for organizational assets. Quick start (standard single-geo tenant) Create a SharePoint site to host Organizational Assets Libraries (often the Brand Center site). Create three document libraries (typical): ImageAssets, DocumentAssets (templates), FontAssets. Grant your intended audience Read access (commonly Everyone except external users via the site’s Visitors group). Enable the SharePoint Online Public CDN (tenant setting). Add a Public CDN origin for each library path (one origin per library). Upload approved assets (images, templates, fonts) into their respective libraries. Register each library with Add-SPOOrgAssetsLibrary (repeat per library). Validate registration and end-user experience, then allow up to 24 hours for Office apps to reflect changes. If you’re Multi-Geo or using Information Barriers: follow the same flow, but repeat per geo and complete registration while the site is in Open IB mode (details below). Key constraints and gotchas Multi-Geo: plan a repeatable per-geo pattern (typically one Org Assets site + matching libraries per geo) and keep naming consistent. Information Barriers (IB): Add-SPOOrgAssetsLibrary cannot be run when the target site is segmented—create and register libraries first (site in Open mode), then segment if needed. The “Everyone except external users” principal may be hidden by default, but it’s still commonly used for broad read access. Brand Center: many orgs host Org Assets Libraries in the Brand Center site; if Brand Center is created after libraries exist, it typically detects and uses them automatically. A public CDN must be enabled to support Organizational Assets Libraries. The “Everyone except external users” principal may be hidden by default, but it’s still commonly used for broad read access. Brand Center: many orgs host Org Assets Libraries in the Brand Center site; if Brand Center is created after libraries exist, it typically detects and uses them automatically. A public CDN must be enabled to support Organizational Assets Libraries. Implementation steps Prerequisites: SharePoint Online Management Shell access (or equivalent), permission to manage tenant settings, and the ability to create sites and libraries in each geo. Create a site to host your Organizational Assets Libraries (many orgs use a communication site). For ease of support, keep the site name, library names, and structure consistent over time. Note: A Communication site is recommended, but a Team site can also work. Example site URLs: In a standard tenant you’ll have one site; in Multi-Geo you’ll typically use one per geo. Primary geo: https://contoso.sharepoint.com/sites/BrandCenter EUR geo: https://contosoEUR.sharepoint.com/sites/BrandCenter APC geo: https://contosoAPC.sharepoint.com/sites/BrandCenter If your tenant uses Information Barriers, keep each site in Open IB mode while creating the Org Assets Libraries. You can segment the site later (if required) after libraries are created. Configure a public CDN (required) To use Brand Center and Organizational Assets Libraries, configure SharePoint Online to use a Public CDN. Set-SPOTenantCdnEnabled -CdnType Public -Enable $true Example output: Public CDN enabled locations: SITES/BRANDCENTER/FONTS */MASTERPAGE (configuration pending) */STYLE LIBRARY (configuration pending) */CLIENTSIDEASSETS (configuration pending) Note: You will see the new CDN is in a pending state until complete. This will take some time. Wait for the CDN to finish provisioning. Re-run the status/list commands until “pending” entries clear. Get-SPOTenantCdnEnabled -CdnType Public Get-SPOTenantCdnOrigins -CdnType Public Add CDN origins for each library Add allowed CDN origins for each asset library path (typically one origin per library). Example: Add-SPOTenantCdnOrigin -OriginUrl sites/BrandCenter/ImageAssets -CdnType Public Add-SPOTenantCdnOrigin -OriginUrl sites/BrandCenter/TemplateAssets -CdnType Public Add-SPOTenantCdnOrigin -OriginUrl sites/BrandCenter/FontAssets -CdnType Public Set permissions (required for broad consumption) To ensure most users can consume the assets, grant Everyone except external users (often abbreviated as EEEU) Read access (commonly via the site’s Visitors group). Example: add Everyone except external users to the Visitors group of the Organizational Assets site. Connect-SPOService -Url 'https://contoso-admin.sharepoint.com' $tenant = "9cfc42cb-51da-4055-87e9-b20a170b6ba3" $site = Get-SPOSite -Identity "https://contoso.sharepoint.com/sites/BrandCenter" $group = Get-SPOSiteGroup $site -Group "BrandCenter Visitors" Add-SPOUser -LoginName ("c:0-.f|rolemanager|spo-grid-all-users/" + $tenant) -Site $site -Group $group.Title Note: Organizational Assets Libraries respect SharePoint security trimming. If you need a narrower audience, grant Read to the appropriate groups instead of tenant-wide access. In many environments, Everyone except external users is required during registration (Add-SPOOrgAssetsLibrary) so Office can enumerate the library—test and confirm in your tenant before removing broad access. Create libraries and upload assets Create a document library for each asset type you plan to publish (for example: images, Office templates, fonts). For example: Upload your assets into the appropriate libraries. Example: Register each library using Add-SPOOrgAssetsLibrary. For this to work, Everyone except external users must already have access to the site (for example, via the Visitors group). Office Template Library Example: Add-SPOOrgAssetsLibrary -LibraryUrl 'https://contoso.sharepoint.com/sites/BrandCenter/DocumentAssets' -OrgAssetType OfficeTemplateLibrary Image Document Library Example: Add-SPOOrgAssetsLibrary -LibraryUrl 'https://contoso.sharepoint.com/sites/BrandCenter/ImageAssets' -OrgAssetType ImageDocumentLibrary Font Document Library Example: Add-SPOOrgAssetsLibrary -LibraryUrl 'https://contoso.sharepoint.com/sites/BrandCenter/FontAssets' -OrgAssetType OfficeFontLibrary -CdnType Public Optional: Enable Copilot support for an image library (only applicable to ImageDocumentLibrary). Set-SPOOrgAssetsLibrary -LibraryUrl 'https://contoso.sharepoint.com/sites/BrandCenter/ImageAssets' -OrgAssetType ImageDocumentLibrary -CopilotSearchable $true Multi-Geo mini runbook (recommended pattern) Use this as a simple tracking sheet so each geo ends up with a complete, consistent setup. Geo Site URL Libraries CDN origins added Libraries registered Primary https://<tenant>.sharepoint.com/sites/<BrandCenterOrAssetsSite> ImageAssets / DocumentAssets / FontAssets Yes/No Yes/No EUR https://<tenant>EUR.sharepoint.com/sites/<BrandCenterOrAssetsSite> ImageAssets / DocumentAssets / FontAssets Yes/No Yes/No APC https://<tenant>APC.sharepoint.com/sites/<BrandCenterOrAssetsSite> ImageAssets / DocumentAssets / FontAssets Yes/No Yes/No Naming standard (strongly recommended): keep the same site path and the same library names in every geo (for example, always ImageAssets, DocumentAssets, FontAssets). This minimizes per-geo scripting differences and reduces support effort. Wrap-up At this point, each geo should have its own site, libraries, CDN origins, and registered Organizational Assets Libraries. From here, focus on governance (who can publish/approve assets), naming standards, and ongoing lifecycle management (retire old templates/fonts and keep branding current). Validate configuration Admin checks (PowerShell) Confirm the Public CDN is enabled. Confirm CDN origins include one entry per assets library path. List registered Org Assets Libraries and verify each URL + type is present. Get-SPOTenantCdnEnabled -CdnType Public Get-SPOTenantCdnOrigins -CdnType Public Get-SPOOrgAssetsLibrary End-user checks (Office apps) In PowerPoint/Word, confirm organizational templates appear in the template picker (if you registered an OfficeTemplateLibrary). In Office font lists, confirm your org fonts appear (if you registered an OfficeFontLibrary). For image libraries, confirm approved brand images appear in supported pickers; if you enabled -CopilotSearchable, confirm images are discoverable as expected. Timing: New registrations and updates can take up to 24 hours to appear in Office apps. If you updated content, run Set-SPOOrgAssetsLibrary for each changed library, then wait for propagation. Updating content in existing Org Assets Libraries If you already have Organizational Assets Libraries registered and you need to publish updated templates, fonts, or images, use the process below. The high-level flow is: update content → run Set-SPOOrgAssetsLibrary (per library) → wait for propagation. Replace or update content in each library. Upload the new versions of templates/fonts/images into the appropriate library (and remove/retire older versions if needed). If Multi-Geo applies, repeat per geo. Update the matching libraries in each geo’s site so users in each geo get the same (or intentionally regional) set of assets. Run Set-SPOOrgAssetsLibrary for each updated library. Execute the cmdlet against the library URL to refresh the configuration after content changes (run it once per library you updated). Wait for Office app propagation. Allow up to 24 hours for updates to begin showing in Office apps. Example: Set-SPOOrgAssetsLibrary -LibraryUrl 'https://contoso.sharepoint.com/sites/BrandCenter/DocumentAssets' -OrgAssetType OfficeTemplateLibrary Notes: If your site is segmented by Information Barriers, confirm the cmdlet behavior in your environment before making changes, and prefer performing registration/updates while the site is in Open mode when possible. For image libraries, if you are using Copilot integration settings (for example -CopilotSearchable), keep the setting consistent when you run Set-SPOOrgAssetsLibrary. Make sure the intended audience still has Read access to the site/library; otherwise users may not see updates due to security trimming. Please note: After registering (or updating) your assets libraries, it can take up to 24 hours before changes become available in Office apps. Once fully enabled, Office apps will surface your templates and fonts. Below is an example. Example of interacting with Org Assets from M365 Apps Org Fonts from PowerPoint: From SharePoint: From Office Apps: Troubleshooting tips If Add-SPOOrgAssetsLibrary fails, confirm the site is not segmented by Information Barriers (Open mode during setup). If assets don’t appear in Office apps, wait for propagation (up to 24 hours) and re-check that the library was registered successfully. If CDN commands show “pending”, allow time for provisioning and re-run the status command. If users can’t see assets, verify the site/library permissions include Everyone except external users (or the intended audience group). Guidance: Using the SharePoint Online Public CDN Enabling the SharePoint Online Public CDN is a required and supported configuration for Organizational Assets Libraries, Brand Center, and related Office experiences. While the word “public” can sound concerning, it’s important to understand what is (and is not) exposed. We take great care to protect the data that runs your business. Data stored in the Microsoft 365 CDN is encrypted both in transit and at rest, and access to data in the Microsoft 365 SharePoint CDN is secured by Microsoft 365 user permissions and token authorization. Requests for data in the Microsoft 365 SharePoint CDN must be referred (redirected) from your Microsoft 365 tenant or an authorization token won't be generated. See: Content delivery networks - Microsoft 365 Enterprise | Microsoft Learn What “Public CDN” actually means Only explicitly approved library paths are cached The CDN does not expose your entire tenant. Administrators must explicitly register CDN origins (specific library paths). If a library is not registered as a CDN origin, it is not served via the CDN. No new content types are exposed The CDN is intended for static, non-sensitive assets such as: Brand images Office templates Fonts It is not designed for documents containing confidential or regulated data. Why Microsoft requires a Public CDN for Org Assets? Performance and reliability Office clients worldwide retrieve assets faster using geographically distributed edge caching. This avoids repeated downloads from SharePoint origin sites. Consistent Office app experiences PowerPoint, Word, Excel, and Copilot rely on CDN-backed delivery to surface: Templates Fonts Brand images Without a public CDN, these features may not function correctly or at all. Best practices Use the practices below to keep Organizational Assets Libraries reliable, secure, and easy for end users to adopt. Where relevant, notes call out additional considerations for Multi-Geo, Information Barriers, Brand Center, and Copilot. Governance and ownership checklist Owners/publishers: named group who can add/change assets (limited membership). Approvals: defined review/approval step before publishing new templates/fonts/images. Versioning/retention: how you retire old assets and prevent outdated branding from appearing in pickers. Rollback plan: how to revert a bad template/font/image quickly. Change communication: how you notify users about new/updated assets and expected timing (up to 24 hours). Assign clear owners (typically Brand/Comms) and a small admin group (typically IT) for each geo’s library and site. Decide what is “approved” vs “draft” content, and enforce it with a simple publishing process (for example, a review checklist or an approvals flow). Version and retire assets deliberately: keep one “current” template set and archive old assets to prevent users from picking outdated branding. Information architecture and naming Keep library names and structures consistent across geos (same library names, same folder conventions) to simplify support and documentation. Use descriptive filenames users can recognize in pickers (for example, “Contoso_Proposal_Template_v3”). Prefer a small number of clearly defined libraries by asset type (images, templates, fonts) rather than many small libraries. Permissions and access Ensure your intended audience has at least Read access to the site and libraries; Organizational Assets still follow SharePoint security trimming. If you use broad access (for example, Everyone except external users), document it and pair it with tight contributor permissions so only approved publishers can change assets. Avoid breaking inheritance in ways that make troubleshooting difficult—keep permissions simple and predictable whenever possible. CDN configuration Plan CDN changes ahead of time: enabling and provisioning can take time, and changes may not be immediate. Register only the origins you need (one per assets library path) and keep them consistent across environments. After changes, allow for propagation time before validating in Office apps. Multi-Geo and Brand Center Use a repeatable pattern: one site + matching libraries per geo, with the same structure and operational runbook. Be aware Brand Center is created in the primary geo; confirm how your org wants to manage global vs regional assets. Document which assets are global (shared everywhere) vs regional (geo-specific) to avoid confusion for publishers and users. Information Barriers (IB) sequencing Create and register Org Assets Libraries before segmenting the site when IB is enabled (create while the site is in Open mode, then segment later if required). After segmentation, re-validate that the right audience can still read the libraries (and that publishers can still manage content). Copilot readiness (image libraries) Use consistent, high-quality metadata for images (titles, descriptions, and tags). Copilot search quality depends heavily on this. If enabling image tagging integration, standardize on a tagging vocabulary (for example, brand terms, campaigns, departments, regions) so results are predictable. Only enable Copilot searchable settings on libraries where content is approved and intended for broad reuse. Q&A Q: What is an Organizational Assets Library (OAL)? A: It’s a SharePoint document library (or set of libraries) that you register so Office apps can surface approved templates, fonts, and images to users directly within the app experience. Q: Do I need SharePoint Brand Center to use OAL? A: No. You can use Organizational Assets Libraries without Brand Center. Brand Center can make asset management more accessible, for example, allowing SharePoint sites to use organizational branding, but OAL can be configured on its own. Q: Why is a “Public CDN” required, and is it safe? A: Office experiences rely on CDN-backed delivery for performance and reliability. “Public CDN” does not mean your whole tenant is exposed—only the specific library paths you register as CDN origins are cached. Access is still governed by Microsoft 365 authentication, token authorization, and SharePoint permissions. Q: Can I use this guide in a standard (single-geo) tenant? A: Yes. In a standard tenant you usually create one site and one set of libraries. The Multi-Geo guidance is only needed if your tenant is Multi-Geo (in which case you’ll typically repeat the pattern per geo). Q: How do Information Barriers (IB) affect setup? A: If a site is segmented, Add-SPOOrgAssetsLibrary cannot register the library. Create the site and register the libraries while the site is in Open mode, then segment afterward if required. Q: Why does “Everyone except external users” (EEEU) matter? A: In many environments, EEEU is required during library registration so Office can enumerate the library. However, OAL still respects SharePoint security trimming. If broad internal availability is the goal, a common pattern is to grant EEEU Read (often via the Visitors group) so Office apps can surface the assets to most internal users. If you need a narrower audience, use a group instead. Q: How long until assets show up (or update) in Office apps? A: It can take up to 24 hours for new registrations or updates to propagate. If you replaced content in an existing library, run Set-SPOOrgAssetsLibrary for each updated library, then allow time for Office apps to refresh. Q: How do I update content in an existing Org Assets Library? A: Replace the files in the library (and repeat across geos if applicable), then run Set-SPOOrgAssetsLibrary against each library you updated. After that, allow up to 24 hours for the updated assets to start showing in Office apps. Q: Do I need to run Set-SPOOrgAssetsLibrary every time I replace files? A: If you want Office apps to reliably pick up changes, run Set-SPOOrgAssetsLibrary after you update content (especially when publishing new/updated templates, fonts, or images). Treat it as the “refresh” step, then wait for propagation. Q: When should I enable Copilot support (CopilotSearchable) for an image library? A: Enable it only for libraries that contain approved, broadly reusable images and have strong metadata (title/description/tags). This helps ensure search results are on-brand and reduces the chance of surfacing unreviewed content. Q: Can I undo this later? A: Yes. You can unregister an Organizational Assets Library using SharePoint Online PowerShell (for example, Remove-SPOOrgAssetsLibrary) and remove CDN origins if you no longer need them. Plan governance so you can retire assets cleanly without disrupting users. Q: Users can’t see the assets (or updates)—what should I check first? A: Start with (1) permissions to the site/library (security trimming), (2) successful registration via Add-SPOOrgAssetsLibrary, (3) if you’re expecting an update, confirm you ran Set-SPOOrgAssetsLibrary for that library, (4) CDN provisioning status and configured origins, and (5) propagation time (up to 24 hours). Additional Reading Create an organization assets library - SharePoint in Microsoft 365 | Microsoft Learn Connect organizational asset libraries to Copilot for an on-brand experience - SharePoint in Microsoft 365 | Microsoft Learn Connect organizational asset libraries to PowerPoint for an on-brand experience - SharePoint in Microsoft 365 | Microsoft Learn Set up and connect organizational asset library (OAL) with image tagging to Copilot search | Microsoft Learn Add-SPOOrgAssetsLibrary (Microsoft.Online.SharePoint.PowerShell) | Microsoft Learn SharePoint Brand Center - SharePoint in Microsoft 365 | Microsoft Learn How to Enable Enterprise Brand Images with PowerPoint Copilot - SharePoint in Microsoft 365 | Microsoft Learn Office 365 Content Delivery Network (CDN) Quickstart - Microsoft 365 Enterprise | Microsoft Learn Use Office 365 Content Delivery Network (CDN) with SharePoint Online - Microsoft 365 Enterprise | Microsoft Learn Content delivery networks - Microsoft 365 Enterprise | Microsoft Learn Multi-Geo Capabilities in OneDrive and SharePoint - Microsoft 365 Enterprise | Microsoft Learn Use Information Barriers with SharePoint | Microsoft Learn
Comparing Open-Source vs Closed LLMs for Enterprise Apps
LLMs are advanced AI models trained in vast data. They enable tasks such as summarization, translation, content creation, and data analysis. When companies build applications that use AI, one of the most important decisions they face is choosing the right type of Large Language Model (LLM). There are two main choices: open‑source LLMs and closed or proprietary LLMs. Understanding the differences between them helps businesses decide which option fits their needs, goals, and security requirements.Preparing for Azure PostgreSQL Certificate Authority Rotation: A Comprehensive Operational Guide
The Challenge It started with a standard notification in the Azure Portal: Tracking-ID YK3N-7RZ. A routine Certificate Authority (CA) rotation for Azure Database for PostgreSQL. As Cloud Solution Architects, we’ve seen this scenario play out many times. The moment “certificate rotation” is mentioned, a wave of unease ripples through engineering teams. Let’s be honest: for many of us—ourselves included—certificates represent the edge of our technical “comfort zone.” We know they are critical for security, but the complexity of PKI chains, trust stores, and SSL handshakes can be intimidating. There is a silent fear: “If we touch this, will we break production?” We realized we had a choice. We could treat this as an opportunity, and we could leave that comfort zone. We approached our customer with a proactive proposal: Let’s use this event to stop fearing certificates and start mastering them. Instead of just patching the immediate issue, we used this rotation as a catalyst to review and upgrade the security posture of their database connections. We wanted to move from “hoping it works” to “knowing it’s secure.” The response was overwhelmingly positive. The teams didn’t just want a quick fix; they wanted “help for self-help.” They wanted to understand the mechanics behind sslmode and build the confidence to manage trust stores proactively. This guide is the result of that journey. It is designed to help you navigate the upcoming rotation not with anxiety, but with competence—turning a mandatory maintenance window into a permanent security improvement. Two Levels of Analysis A certificate rotation affects your environment on two distinct levels, requiring different expertise and actions: Level Responsibility Key Questions Actions Platform Level Cloud/Platform Teams Which clusters, services, and namespaces are affected? How do we detect at scale? Azure Service Health monitoring, AKS scanning, infrastructure-wide assessment Application Level Application/Dev Teams What SSL mode? Which trust store? How to update connection strings? Code changes, dependency updates, trust store management This article addresses both levels - providing platform-wide detection strategies (Section 5) and application-specific remediation guidance (Platform-Specific Remediation). Business Impact: In production environments, certificate validation failures cause complete database connection outages. A single missed certificate rotation has caused hours of downtime for enterprise customers, impacting revenue and customer trust. Who’s Affected: DevOps engineers, SREs, database administrators, and platform engineers managing Azure PostgreSQL instances - especially those using: - Java applications with custom JRE cacerts - Containerized workloads with baked-in trust stores - Strict SSL modes (sslmode=verify-full, verify-ca) The Solution What we’ll cover: 🛡️ Reliability: How to prevent database connection outages through proactive certificate management 🔄 Resiliency: Automation strategies that ensure your trust stores stay current 🔒 Security: Maintaining TLS security posture while rotating certificates safely Key Takeaway: This rotation is a client trust topic, not a server change. Applications trusting root CAs (DigiCert Global Root G2, Microsoft RSA Root CA 2017) without intermediate pinning are unaffected. Risk concentrates where strict validation meets custom trust stores. 📦 Platform-Specific Implementation: Detailed remediation guides for Java, .NET, Python, Node.js, and Kubernetes are available in our GitHub Repository. Note: The GitHub Repository. contains community-contributed content provided as-is. Test all scripts in non-production environments before use. 1. Understanding Certificate Authority Rotation What Changes During CA Rotation? Azure Database for PostgreSQL uses TLS/SSL to encrypt client-server connections. The database server presents a certificate chain during the TLS handshake: Certificate Chain Structure: Figure: Certificate chain structure showing the rotation from old intermediate (red, deprecated) to new intermediate (blue, active after rotation). Client applications must trust the root certificates (green) to validate the chain. 📝 Diagram Source: The Mermaid source code for this diagram is available in certificate-chain-diagram.mmd. Why Root Trust Matters Key Principle: If your application trusts the root certificate and allows the chain to be validated dynamically, you are not affected. The risk occurs when: Custom trust stores contain only the old intermediate certificate (not the root) Certificate pinning is implemented at the intermediate level Strict validation is enabled (sslmode=verify-full in PostgreSQL connection strings) 2. Who Is Affected and Why Risk Assessment Matrix Application Type Trust Store SSL Mode Risk Level Action Required Cloud-native app (Azure SDK) OS Trust Store require 🟢 Low None - Azure SDK handles automatically Java app (default JRE) System cacerts verify-ca 🟡 Medium Verify JRE version (11.0.16+, 17.0.4+, 8u381+) Java app (custom cacerts) Custom JKS file verify-full 🔴 High Update custom trust store with new intermediate .NET app (Windows) Windows Cert Store require 🟢 Low None - automatic via Windows Update Python app (certifi) certifi bundle verify-ca 🟡 Medium Update certifi package (pip install --upgrade certifi) Node.js app (default) Built-in CAs verify-ca 🟢 Low None - Node.js 16+, 18+, 20+ auto-updated Container (Alpine) /etc/ssl/certs verify-full 🔴 High Update base image or install ca-certificates-bundle Container (custom) Baked-in certs verify-full 🔴 High Rebuild image with updated trust store How to Read This Matrix Use the above matrix to quickly assess whether your applications are affected by CA rotation. Here is an overview, how you read the matrix: Column Meaning Application Type What kind of application do you have? (e.g., Java, .NET, Container) Trust Store Where does the application store its trusted certificates? SSL Mode How strictly does the application validate the server certificate? Risk Level 🟢 Low / 🟡 Medium / 🔴 High - How likely is a connection failure? Action Required What specific action do you need to take? Risk Level Logic: Risk Level Why? 🟢 Low Automatic updates (OS/Azure SDK) or no certificate validation 🟡 Medium Manual update required but straightforward (e.g., pip install --upgrade certifi) 🔴 High Custom trust store must be manually updated - highest outage risk SSL Mode Security Posture Understanding SSL modes is critical because they determine both security posture AND rotation impact. This creates a dual consideration: SSL Mode Certificate Validation Rotation Impact Security Level Recommendation disable ❌ None ✅ No impact 🔴 INSECURE Never use in production allow ❌ None ✅ No impact 🟠 WEAK Not recommended prefer ❌ Optional ✅ Minimal 🟡 WEAK Not recommended require ❌ No (Npgsql 6.0+) ✅ No impact 🟡 WEAK Upgrade to verify-full verify-ca ✅ Chain only 🔴 Critical 🔵 MODERATE Update trust stores verify-full ✅ Chain + hostname 🔴 Critical 🟢 SECURE Recommended - Update trust stores Key Insight: Applications using weak SSL modes (everything below verify-ca) are technically unaffected by CA rotation but represent security vulnerabilities. The safest path is verify-full with current trust stores. ⚖️ The Security vs. Resilience Trade-off The Paradox: Secure applications (verify-full) have the highest rotation risk 🔴, while insecure applications (require) are unaffected but have security gaps. Teams discovering weak SSL modes during rotation preparation face a critical decision: Option Approach Rotation Impact Security Impact Recommended For 🚀 Quick Fix Keep weak SSL mode (require) ✅ No action needed ⚠️ Security debt remains Emergency situations only 🛡️ Proper Fix Upgrade to verify-full 🔴 Requires trust store updates ✅ Improved security posture All production systems Our Recommendation: Use CA rotation events as an opportunity to improve your security posture. The effort to update trust stores is a one-time investment that pays off in long-term security. Common Scenarios Scenario 1: Enterprise Java Application Problem: Custom trust store created 2+ years ago for PCI compliance Risk: High - contains only old intermediate certificates Solution: Export new intermediate from Azure, import to custom cacerts Scenario 2: Kubernetes Microservices Problem: Init container copies trust store from ConfigMap at startup Risk: High - ConfigMap never updated since initial deployment Solution: Update ConfigMap, redeploy pods with new trust store Scenario 3: Legacy .NET Application Problem: .NET Framework 4.6 on Windows Server 2016 (no Windows Update) Risk: Medium - depends on manual certificate store updates Solution: Import new intermediate to Windows Certificate Store manually 3. Trust Store Overview A trust store is the collection of root and intermediate CA certificates that your application uses to validate server certificates during TLS handshakes. Understanding where your application’s trust store is located determines how you’ll update it for CA rotations. Trust Store Locations by Platform Category Platform Trust Store Location Update Method Auto-Updated? OS Level Windows Cert:\LocalMachine\Root Windows Update ✅ Yes Debian/Ubuntu /etc/ssl/certs/ca-certificates.crt apt upgrade ca-certificates ✅ Yes (with updates) Red Hat/CentOS /etc/pki/tls/certs/ca-bundle.crt yum update ca-certificates ✅ Yes (with updates) Runtime Level Java JRE $JAVA_HOME/lib/security/cacerts Java security updates ✅ With JRE updates Python (certifi) site-packages/certifi/cacert.pem pip install --upgrade certifi ❌ Manual Node.js Bundled with runtime Node.js version upgrade ✅ With Node.js updates Custom Custom JKS Application-specific path keytool -importcert ❌ Manual Container image /etc/ssl/certs (baked-in) Rebuild container image ❌ Manual ConfigMap mount Kubernetes ConfigMap Update ConfigMap, redeploy ❌ Manual Why This Matters for CA Rotation Applications using auto-updated trust stores (OS-managed, current runtime versions) generally handle CA rotations automatically. The risk concentrates in: Custom trust stores created for compliance requirements (PCI-DSS, SOC 2) that are rarely updated Baked-in container certificates from images built months or years ago Outdated runtimes (old JRE versions, frozen Python environments) that haven’t received security updates Air-gapped environments where automatic updates are disabled When planning for CA rotation, focus your assessment efforts on applications in the “Manual” update category. 4. Platform-Specific Remediation 📦 Detailed implementation guides are available in our GitHub repository: azure-certificate-rotation-guide Quick Reference: Remediation by Platform Platform Trust Store Location Update Method Guide Java $JAVA_HOME/lib/security/cacerts Update JRE or manual keytool import java-cacerts.md .NET (Windows) Windows Certificate Store Windows Update (automatic) dotnet-windows.md Python certifi package pip install --upgrade certifi python-certifi.md Node.js Built-in CA bundle Update Node.js version nodejs.md Containers Base image /etc/ssl/certs Rebuild image or ConfigMap containers-kubernetes.md Scripts & Automation Script Purpose Download State Scan-AKS-TrustStores.ps1 Scan all pods in AKS for trust store configurations PowerShell tested validate-connection.sh Test PostgreSQL connection with SSL validation Bash not tested update-cacerts.sh Update Java cacerts with new intermediate Bash not tested 5. Proactive Detection Strategies Database-Level Discovery: Identifying Connected Clients One starting point for impact assessment is querying the PostgreSQL database itself to identify which applications are connecting. We developed a SQL query that joins pg_stat_ssl with pg_stat_activity to reveal active TLS connections, their SSL version, and cipher suites. 🔍 Get the SQL Query: Download the complete detection script from our GitHub repository: detect-clients.sql Important Limitations This query has significant constraints that you must understand before relying on it for CA rotation planning: Limitation Impact Mitigation Point-in-time snapshot Only shows currently connected clients Run query repeatedly over days/weeks to capture periodic jobs and batch processes No certificate details Cannot identify which CA certificate the client is using Requires client-side investigation (trust store analysis) Connection pooling May show pooler instead of actual application Use application_name in connection strings to identify true source Idle connections Long-running connections may be dormant Cross-reference with application activity logs Recommended approach: Use this query to create an initial inventory, then investigate each unique application_name and client_addr combination to determine their trust store configuration and SSL mode. Proactive Monitoring with Azure Monitor To detect certificate-related issues before and after CA rotation, configure Azure Monitor alerts. This enables early warning when SSL handshakes start failing. Why this matters: After CA rotation, applications with outdated trust stores will fail to connect. An alert allows you to detect affected applications quickly rather than waiting for user reports. Official Documentation: For complete guidance on creating and managing alerts, see Azure Monitor Alerts Overview and Create a Log Search Alert. Here is a short example of an Azure Monitor Alert definition as a starting point. { "alertRule": { "name": "PostgreSQL SSL Connection Failures", "severity": 2, "condition": { "query": "AzureDiagnostics | where ResourceType == 'SERVERS' and Category == 'PostgreSQLLogs' and Message contains 'SSL error' | summarize count() by bin(TimeGenerated, 5m)", "threshold": 5, "timeAggregation": "Total", "windowSize": "PT5M" } } } Alert Configuration Notes: Setting Recommended Value Rationale Severity 2 (Warning) Allows investigation without triggering critical incident response Threshold 5 failures/5min Filters noise while catching genuine issues Evaluation Period 5 minutes Balances responsiveness with alert fatigue Action Group Platform Team Ensures quick triage and coordination 6. Production Validation Pre-Rotation Validation Checklist Inventory all applications connecting to Azure PostgreSQL Identify trust store locations for each application Verify root certificate presence in trust stores Test connection with new intermediate in non-production environment Update monitoring alerts for SSL connection failures Prepare rollback plan if issues occur Schedule maintenance window (if required) Notify stakeholders of potential impact Testing Procedure We established a systematic 3-step validation process to ensure zero downtime. This approach moves from isolated testing to gradual production rollout. 🧪 Technical Validation Guide: For the complete list of psql commands, connection string examples for Windows/Linux, and automated testing scripts, please refer to our Validation Guide in the GitHub repository. Connection Testing Strategy The core of our validation strategy was testing connections with explicit sslmode settings. We used the psql command-line tool to simulate different client behaviors. Test Scenario Purpose Expected Result Encryption only (sslmode=require) Verify basic connectivity Connection succeeds even with unknown CA CA validation (sslmode=verify-ca) Verify trust store integrity Connection succeeds only if CA chain is valid Full validation (sslmode=verify-full) Verify strict security compliance Connection succeeds only if CA chain AND hostname match Pro Tip: Test with verify-full and an explicit root CA file containing the new Microsoft/DigiCert root certificates before the rotation date. This validates that your trust stores will work after the intermediate certificate changes. Step 1: Test in Non-Production Validate connections against a test server using the new intermediate certificate (Azure provides test endpoints during the rotation window). Step 2: Canary Deployment Deploy the updated trust store to a single “canary” instance or pod. Monitor: - Connection success rate - Error logs - Response times Step 3: Gradual Rollout Once the canary is stable, proceed with a phased rollout: 1. Update 10% of pods 2. Monitor for 1 hour 3. Update 50% of pods 4. Monitor for 1 hour 5. Complete rollout 7. Best Practices and Lessons Learned Certificate Management Best Practices Practice Guidance Example Trust Root CAs, Not Intermediates Configure trust stores with root CA certificates only. This provides resilience against intermediate certificate rotations. Trust Microsoft TLS RSA Root G2 and DigiCert Global Root G2 instead of specific intermediates Automate Trust Store Updates Use OS-provided trust stores when possible (automatically updated). For custom trust stores, implement CI/CD pipelines. Schedule bi-annual trust store audits Use SSL Mode Appropriately Choose SSL mode based on security requirements. verify-ca is recommended for most scenarios. See Security Posture Matrix in Section 2 Maintain Container Images Rebuild container images monthly to include latest CA certificates. Use init containers for runtime updates. Multi-stage builds with CA certificate update step Avoid Certificate Pinning Never pin intermediate certificates. If pinning is required for compliance, implement automated update processes. Pin only root CA certificates if absolutely necessary SSL Mode Decision Guide SSL Mode Security Level Resilience When to Use require Medium High Encrypted traffic without certificate validation. Use when CA rotation resilience is more important than MITM protection. verify-ca High Medium Validates certificate chain. Recommended for most production scenarios. verify-full Highest Low Strictest validation with hostname matching. Use only when compliance requires it. Organizational Communication Model Effective certificate rotation requires structured communication across multiple layers: Layer Responsibility Key Action Azure Service Health Microsoft publishes announcements to affected subscriptions Monitor Azure Service Health proactively Platform/Cloud Team Receives Azure announcements, triages criticality Follow ITSM processes, assess impact Application Teams Execute application-level changes Update trust stores, validate connections Security Teams Define certificate validation policies Set compliance requirements Ownership and Responsibility Matrix Team Responsibility Deliverable Platform/Cloud Team Monitor Azure Service Health, coordinate response Impact assessment, team notifications Application Teams Application-level changes (connection strings, trust stores) Updated configurations, validation results Security Teams Define certificate policies, compliance requirements Policy documentation, audit reports All Teams (Shared) Certificate lifecycle collaboration Playbooks, escalation paths, training Certificate Rotation Playbook Components Organizations should establish documented playbooks including: Component Recommended Frequency Purpose Trust Store Audits Bi-annual (every 6 months) Ensure certificates are current Certificate Inventory Quarterly review Know what certificates exist where Playbook Updates Annual or after incidents Keep procedures current Team Training Annual Build knowledge and confidence Field Observations: Common Configuration Patterns Pattern Observation Risk Implicit SSL Mode Teams don’t explicitly set sslmode, relying on framework defaults Unexpected behavior during CA rotation Copy-Paste Configurations Connection strings copied without understanding options Works until certificate changes expose gaps Framework-Specific Defaults Java uses JRE trust store, .NET uses Windows Certificate Store, Python depends on certifi package Some require manual updates, some are automatic Framework Trust Store Defaults Framework Default Trust Store Update Method Risk Level Java/Quarkus JRE cacerts Manual or JRE update Medium - requires awareness .NET Windows Certificate Store Windows Update Low - automatic Node.js Bundled certificates Node.js version update Low - automatic Python certifi package pip install --upgrade certifi High - manual intervention required Knowledge and Confidence Challenges Challenge Impact Mitigation Limited certificate knowledge Creates uncertainty and risk-averse behavior Proactive education, hands-on workshops Topic intimidation “Certificates” can seem complex, leading to avoidance Reality: Implementation is straightforward once understood Previous negative experiences Leadership concerns based on past incidents Document successes, share lessons learned Visibility gaps Lack of visibility into application dependencies Maintain certificate inventory, use discovery tools Monitoring Strategy (Recommended for Post-Rotation): While pre-rotation monitoring focuses on inventory, post-rotation monitoring should track: Key Metrics: - Connection failure rates (group by application, SSL error types) - SSL handshake duration (detect performance degradation) - Certificate validation errors (track which certificates fail) - Application error logs (filter for “SSL”, “certificate”, “trust”) Recommended Alerts: - Threshold: >5 SSL connection failures in 5 minutes - Anomaly detection: Connection failure rate increases >50% - Certificate expiry warnings: 30, 14, 7 days before expiration Dashboard Components: - Connection success rate by application - SSL error distribution (validation failures, expired certificates, etc.) - Certificate inventory with expiry dates - Trust store update status across infrastructure These metrics, alerts and thresholds are only starting points and need to be adjusted based on your environment and needs. Post-Rotation Validation and Telemetry Note: This article focuses on preparation for upcoming certificate rotations. Post-rotation metrics and incident data will be collected after the rotation completes and can inform future iterations of this guidance. Recommended Post-Rotation Activities: Here are some thoughts on post-rotation activities that could create more insights on the effectiveness of the preparation. Incident Tracking: After rotation completes, organizations should track: - Production incidents related to SSL/TLS connection failures - Services affected and their business criticality - Mean Time to Detection (MTTD) for certificate-related issues - Mean Time to Resolution (MTTR) from detection to fix Success Metrics to Measure Pre-Rotation Validation: - Number of services inventoried and assessed - Percentage of services requiring trust store updates - Testing coverage (dev, staging, production) Post-Rotation Outcomes: - Zero-downtime success rate (percentage of services with no impact) - Applications requiring emergency patching - Time from rotation to full validation Impact Assessment Telemetry to Collect: - Total connection attempts vs. failures (before and after rotation) - Duration of any service degradation or outages - ustomer-facing impact (user-reported issues, support tickets) - Geographic or subscription-specific patterns Continuous Improvement Post-Rotation Review: - What worked well in the preparation phase? - Which teams or applications were unprepared? - What gaps exist in monitoring or alerting? - How can communication be improved for future rotations? Documentation Updates: - Update playbooks with lessons learned - Refine monitoring queries based on observed patterns - Enhance team training materials - Share anonymized case studies across the organization 8. Engagement & Next Steps Discussion Questions We’d love to hear from the community: What’s your experience with certificate rotations? Have you encountered unexpected connection failures during CA rotation events? Which trust store update method works best for your environment? OS-managed, runtime-bundled, or custom trust stores? How do you handle certificate management in air-gapped environments? What strategies have worked for your organization? Share Your Experience If you’ve implemented proactive certificate management strategies or have lessons learned from CA rotation incidents, we encourage you to: Comment below with your experiences and tips Contribute to the GitHub repository with additional platform guides or scripts Connect with us on LinkedIn to continue the conversation Call to Action Take these steps now to prepare for the CA rotation: Assess your applications - Use the Risk Assessment Matrix (Section 2) to identify which applications use sslmode=verify-ca or verify-full with custom trust stores Import root CA certificates - Add DigiCert Global Root G2 and Microsoft RSA Root CA 2017 to your trust stores Upgrade SSL mode - Change your connection strings to at least sslmode=verify-ca (recommended: verify-full) for improved security Document your changes - Record which applications were updated, what trust stores were modified, and the validation results Automate for the future - Implement proactive certificate management so future CA rotations are handled automatically (OS-managed trust stores, CI/CD pipelines for container images, scheduled trust store audits) 9. Resources Official Documentation Azure PostgreSQL: Azure PostgreSQL SSL/TLS Concepts Azure PostgreSQL - Connect with TLS/SSL PostgreSQL & libpq: PostgreSQL libpq SSL Support - SSL mode options and environment variables PostgreSQL psql Reference - Command-line tool documentation PostgreSQL Server SSL/TLS Configuration Certificate Authorities: DigiCert Root Certificates Microsoft PKI Repository Microsoft Trusted Root Program Community Resources Let’s Encrypt Root Expiration (2021 Incident) NIST SP 800-57: Key Management Guidelines OWASP Certificate Pinning Cheat Sheet Neon Blog: PostgreSQL Connection Security Defaults Tools and Scripts PowerShell AKS Trust Store Scanner (see Platform-Specific Remediation) PostgreSQL Interactive Terminal (psql) PostgreSQL JDBC SSL Documentation Industry Context Certificate rotation challenges are not unique to Azure PostgreSQL. Similar incidents have occurred across the industry: Historical Incidents: - Let’s Encrypt Root Expiration (2021): Widespread impact when DST Root CA X3 expired, affecting older Android devices and legacy systems - DigiCert Root Transitions: Multiple cloud providers experienced customer impact during CA changes - Internal PKI Rotations: Enterprises face similar challenges when rotating internally-issued certificates Relevant Standards: - NIST SP 800-57: Key Management Guidelines (certificate lifecycle best practices) - OWASP Certificate Pinning: Guidance on balancing security and operational resilience - CIS Benchmarks: Recommendations for TLS/SSL configuration in cloud environments Authors Author Role Contact Andreas Semmelmann Cloud Solution Architect, Microsoft LinkedIn Mpho Muthige Cloud Solution Architect, Microsoft LinkedIn Disclaimers Disclaimer: The information in this blog post is provided for general informational purposes only and does not constitute legal, financial, or professional advice. While every effort has been made to ensure the accuracy of the information at the time of publication, Microsoft makes no warranties or representations as to its completeness or accuracy. Product features, availability, and timelines are subject to change without notice. For specific guidance, please consult your legal or compliance advisor. Microsoft Support Statement: This article represents field experiences and community best practices. For official Microsoft support and SLA-backed guidance: Azure Support: https://azure.microsoft.com/support/ Official Documentation: https://learn.microsoft.com/azure/ Microsoft Q&A: https://learn.microsoft.com/answers/ Production Issues: Always open official support tickets for production-impacting problems. Customer Privacy Notice: This article describes real-world scenarios from customer engagements. All customer-specific information has been anonymized. No NDAs or customer confidentiality agreements were violated in creating this content. AI-generated content disclaimer: This content was generated in whole or in part with the assistance of AI tools. AI-generated content may be incorrect or incomplete. Please review and verify before relying on it for critical decisions. See terms Community Contribution: The GitHub repository referenced in this article contains community-contributed scripts and guides. These are provided as-is for educational purposes and should be tested in non-production environments before use. Tags: #AzurePostgreSQL #CertificateRotation #TLS #SSL #TrustStores #Operations #DevOps #SRE #CloudSecurity #AzureDatabaseAzure AI Foundry vs. Azure Databricks – A Unified Approach to Enterprise Intelligence
Key Insights into Azure AI Foundry and Azure Databricks Complementary Powerhouses: Azure AI Foundry is purpose-built for generative AI application and agent development, focusing on model orchestration and rapid prototyping, while Azure Databricks excels in large-scale data engineering, analytics, and traditional machine learning, forming the data intelligence backbone. Seamless Integration for End-to-End AI: A critical native connector allows AI agents developed in Foundry to access real-time, governed data from Databricks, enabling contextual and data-grounded AI solutions. This integration facilitates a comprehensive AI lifecycle from data preparation to intelligent application deployment. Specialized Roles for Optimal Performance: Enterprises leverage Databricks for its robust data processing, lakehouse architecture, and ML model training capabilities, and then utilize AI Foundry for deploying sophisticated generative AI applications, agents, and managing their lifecycle, ensuring responsible AI practices and scalability. In the rapidly evolving landscape of artificial intelligence, organizations seek robust platforms that can not only handle vast amounts of data but also enable the creation and deployment of intelligent applications. Microsoft Azure offers two powerful, yet distinct, services in this domain: Azure AI Foundry and Azure Databricks. While both contribute to an organization's AI capabilities, they serve different primary functions and are designed to complement each other in building comprehensive, enterprise-grade AI solutions. Decoding the Core Purpose: Foundry for Generative AI, Databricks for Data Intelligence At its heart, the distinction between Azure AI Foundry and Azure Databricks lies in their core objectives and the types of workloads they are optimized for. Understanding these fundamental differences is crucial for strategic deployment and maximizing their combined potential. Azure AI Foundry: The Epicenter for Generative AI and Agents Azure AI Foundry emerges as Microsoft's unified platform specifically engineered for the development, deployment, and management of generative AI applications and AI agents. It represents a consolidation of capabilities from what were formerly Azure AI Studio and Azure OpenAI Studio. Its primary focus is on accelerating the entire lifecycle of generative AI, from initial prototyping to large-scale production deployments. Key Characteristics of Azure AI Foundry: Generative AI Focus: Foundry streamlines the development of large language models (LLMs) and customized generative AI applications, including chatbots and conversational AI. It emphasizes prompt engineering, Retrieval-Augmented Generation (RAG), and agent orchestration. Extensive Model Catalog: It provides access to a vast catalog of over 11,000 foundation models from various publishers, including OpenAI, Meta (Llama 4), Mistral, and others. These models can be deployed via managed compute or serverless API deployments, offering flexibility and choice. Agentic Development: A significant strength of Foundry is its support for building sophisticated AI agents. This includes tools for grounding agents with knowledge, tool calling, comprehensive evaluations, tracing, monitoring, and guardrails to ensure responsible AI practices. Foundry Local further extends this by allowing offline and on-device development. Unified Development Environment: It offers a single management grouping for agents, models, and tools, promoting efficient development and consistent governance across AI projects. Enterprise Readiness: Built-in capabilities such as Role-Based Access Control (RBAC), observability, content safety, and project isolation ensure that AI applications are secure, compliant, and scalable for enterprise use. Figure 1: Conceptual Architecture of Azure AI Foundry illustrating its various components for AI development and deployment. Azure Databricks: The Powerhouse for Data Engineering, Analytics, and Machine Learning Azure Databricks, on the other hand, is an Apache Spark-based data intelligence platform optimized for large-scale data engineering, analytics, and traditional machine learning workloads. It acts as a collaborative workspace for data scientists, data engineers, and ML engineers to process, analyze, and transform massive datasets, and to build and deploy diverse ML models. Key Characteristics of Azure Databricks: Unified Data Analytics Platform: Central to Databricks is its lakehouse architecture, built on Delta Lake, which unifies data warehousing and data lakes. This provides a single platform for data engineering, SQL analytics, and machine learning. Big Data Processing: Excelling in distributed computing, Databricks is ideal for processing large datasets, performing ETL (Extract, Transform, Load) operations, and real-time analytics at scale. Comprehensive ML and AI Workflows: It offers a specialized environment for the full ML lifecycle, including data preparation, feature engineering, model training (both classic and deep learning), and model serving. Tools like MLflow are integrated for tracking, evaluating, and monitoring ML models. Data Intelligence Features: Databricks includes AI-assistive features such as Databricks Assistant and Databricks AI/BI Genie, which enable users to interact with their data using natural language queries to derive insights. Unified Governance with Unity Catalog: Unity Catalog provides a centralized governance solution for all data and AI assets within the lakehouse, ensuring data security, lineage tracking, and access control. Figure 2: The Databricks Data Intelligence Platform with its unified approach to data, analytics, and AI. The Symbiotic Relationship: Integration and Complementary Use Cases While distinct in their primary functions, Azure AI Foundry and Azure Databricks are explicitly designed to work together, forming a powerful, integrated ecosystem for end-to-end AI development and deployment. This synergy is key to building advanced, data-driven AI solutions in the enterprise. Seamless Integration for Enhanced AI Capabilities The integration between the two platforms is a cornerstone of Microsoft's AI strategy, enabling AI agents and generative applications to be grounded in high-quality, governed enterprise data. Key Integration Points: Native Databricks Connector in AI Foundry: A significant development in 2025 is the public preview of a native connector that allows AI agents built in Azure AI Foundry to directly query real-time, governed data from Azure Databricks. This means Foundry agents can leverage Databricks AI/BI Genie to surface data insights and even trigger Databricks Jobs, providing highly contextual and domain-aware responses. Data Grounding for AI Agents: This integration enables AI agents to access structured and unstructured data processed and stored in Databricks, providing the necessary context and knowledge base for more accurate and relevant generative AI outputs. All interactions are auditable within Databricks, maintaining governance and security. Model Crossover and Availability: Foundation models, such as the Llama 4 family, are made available across both platforms. Databricks DBRX models can also appear in the Foundry model catalog, allowing flexibility in where models are trained, deployed, and consumed. Unified Identity and Governance: Both platforms leverage Azure Entra ID for authentication and access control, and Unity Catalog provides unified governance for data and AI assets managed by Databricks, which can then be respected by Foundry agents. Here's a breakdown of how a typical flow might look: Mindmap 1: Illustrates the complementary roles and integration points between Azure Databricks and Azure AI Foundry within an end-to-end AI solution. When to Use Which (and When to Use Both) Choosing between Azure AI Foundry and Azure Databricks, or deciding when to combine them, depends on the specific requirements of your AI project: Choose Azure AI Foundry When You Need To: Build and deploy production-grade generative AI applications and multi-agent systems. Access, evaluate, and benchmark a wide array of foundation models from various providers. Develop AI agents with sophisticated capabilities like tool calling, RAG, and contextual understanding. Implement enterprise-grade guardrails, tracing, monitoring, and content safety for AI applications. Rapidly prototype and iterate on generative AI solutions, including chatbots and copilots. Integrate AI agents deeply with Microsoft 365 and Copilot Studio. Choose Azure Databricks When You Need To: Perform large-scale data engineering, ETL, and data warehousing on a unified lakehouse. Build and train traditional machine learning models (supervised, unsupervised learning, deep learning) at scale. Manage and govern all data and AI assets centrally with Unity Catalog, ensuring data quality and lineage. Conduct complex data analytics, business intelligence (BI), and real-time data processing. Leverage AI-assistive tools like Databricks AI/BI Genie for natural language interaction with data. Require high-performance compute and auto-scaling for data-intensive workloads. Use Both for Comprehensive AI Solutions: The most powerful approach for many enterprises is to leverage both platforms. Azure Databricks can serve as the robust data backbone, handling data ingestion, processing, governance, and traditional ML model training. Azure AI Foundry then sits atop this foundation, consuming the prepared and governed data to build, deploy, and manage intelligent generative AI agents and applications. This allows for: Domain-Aware AI: Foundry agents are grounded in enterprise-specific data from Databricks, leading to more accurate, relevant, and trustworthy AI responses. End-to-End AI Lifecycle: Databricks manages the "data intelligence" part, and Foundry handles the "generative AI application" part, covering the entire spectrum from raw data to intelligent user experience. Optimized Resource Utilization: Each platform focuses on what it does best, leading to more efficient resource allocation and specialized toolsets for different stages of the AI journey. Comparative Analysis: Features and Capabilities To further illustrate their distinct yet complementary nature, let's examine a detailed comparison of their features, capabilities, and typical user bases. Radar Chart 1: This chart visually compares Azure AI Foundry and Azure Databricks across several key dimensions, illustrating their specialized strengths. Azure AI Foundry excels in generative AI and agent orchestration, while Azure Databricks dominates in data engineering, unified data governance, and traditional ML workflows. A Detailed Feature Comparison Feature Category Azure AI Foundry Azure Databricks Primary Focus Generative AI application & agent development, model orchestration Large-scale data engineering, analytics, traditional ML, and AI workflows Data Handling Connects to diverse data sources (e.g., Databricks, Azure AI Search) for grounding AI agents. Not a primary data storage/processing platform. Native data lakehouse architecture (Delta Lake), optimized for big data processing, storage, and real-time analytics. AI/ML Capabilities Foundation models (LLMs), prompt engineering, RAG, agent orchestration, model evaluation, content safety, responsible AI tooling. Traditional ML (supervised/unsupervised), deep learning, feature engineering, MLflow for lifecycle management, Databricks AI/BI Genie. Development Style Low-code agent building, prompt flows, unified SDK/API, templates. Code-first (Python, SQL, Scala, R), notebooks, IDE integrations. Model Access & Deployment Extensive model catalog (11,000+ models), serverless API, managed compute deployments, model benchmarking. Training and serving custom ML models, including deep learning. Models available for deployment through MLflow. Governance & Security Azure-based security & compliance, RBAC, project isolation, content safety guardrails, tracing, evaluations. Unity Catalog for unified data & AI governance, lineage tracking, access control, Entra ID integration. Key Users AI developers, business analysts, citizen developers, AI app builders. Data scientists, data engineers, ML engineers, data analysts. Integration Points Native connector to Databricks AI/BI Genie, Azure AI Search, Microsoft 365, Copilot Studio, Power Platform. Microsoft Fabric, Power BI, Azure AI Foundry, Azure Purview, Azure Monitor, Azure Key Vault. Table 1: A comparative overview of the distinct features and functionalities of Azure AI Foundry and Azure Databricks Concluding Thoughts In essence, Azure AI Foundry and Azure Databricks are not competing platforms but rather essential components of a unified, comprehensive AI strategy within the Azure ecosystem. Azure Databricks provides the robust, scalable foundation for all data engineering, analytics, and traditional machine learning workloads, acting as the "data intelligence platform." Azure AI Foundry then leverages this foundation to specialize in the rapid development, deployment, and operationalization of generative AI applications and intelligent agents. Together, they enable enterprises to unlock the full potential of AI, transforming raw data into powerful, domain-aware, and governed intelligent solutions. Frequently Asked Questions (FAQ) What is the main difference between Azure AI Foundry and Azure Databricks? Azure AI Foundry is specialized for building, deploying, and managing generative AI applications and AI agents, focusing on model orchestration and prompt engineering. Azure Databricks is a data intelligence platform for large-scale data engineering, analytics, and traditional machine learning, built on a Lakehouse architecture. Can Azure AI Foundry and Azure Databricks be used together? Yes, they are designed to work synergistically. Azure AI Foundry can leverage a native connector to access real-time, governed data from Azure Databricks, allowing AI agents to be grounded in enterprise data for more accurate and contextual responses. Which platform should I choose for training large machine learning models? For training large-scale, traditional machine learning, and deep learning models, Azure Databricks is generally the preferred choice due to its robust capabilities for data processing, feature engineering, and ML lifecycle management (MLflow). Azure AI Foundry focuses more on the deployment and orchestration of pre-trained foundation models and generative AI applications. Does Azure AI Foundry replace Azure Machine Learning or Databricks? No, Azure AI Foundry complements these services. It provides a specialized environment for generative AI and agent development, often integrating with data and models managed by Azure Databricks or Azure Machine Learning for comprehensive AI solutions. How do these platforms handle data governance? Azure Databricks utilizes Unity Catalog for unified data and AI governance, providing centralized control over data access and lineage. Azure AI Foundry integrates with Azure-based security and compliance features, ensuring responsible AI practices and data privacy within its generative AI applications.
Diagnose performance issues in Spark jobs through Spark UI.
Agenda Introduction Overview of Spark UI Navigating to Spark UI Jobs Timeline Opening Jobs Timeline Reading Event Timeline Failing Jobs or Executors Diagnosing Failing Jobs Diagnosing Failing Executors Scenario - Memory Issues Scenario - Long Running Jobs Scenario - Identifying Longest Stage Introduction Diagnosing performance issues of job using Spark UI This guide walks you through how to use the Spark UI to diagnose performance issues Overview of Spark UI Job Composition Composed of multiple stages Stages may contain more than one task Task Breakdown Tasks are broken into executors Navigating to Spark UI: Navigating to Cluster's Page Navigate to your cluster’s page: Navigating to Spark UI: Clicking Spark UI Click Spark UI: Jobs Timeline Jobs timeline The jobs timeline is a great starting point for understanding your pipeline or query. It gives you an overview of what was running, how long each step took, and if there were any failures along the way Opening Jobs Timeline Accessing the Jobs Timeline Navigate to the Spark UI Click on the Jobs tab Viewing the Event Timeline Click on Event Timeline Highlighted in red in the screenshot Example Timeline Shows driver and executor 0 being added Failing Jobs or Executors: Example of Failed Job Failed Job Example Indicated by a red status Shown in the event timeline Removed Executors Also indicated by a red status Shown in the event timeline Failing Jobs or Executors: Common Reasons for Executors Being Removed Autoscaling Expected behavior, not an error See Enable autoscaling for more details Compute configuration reference - Azure Databricks | Microsoft Learn Spot instance losses Cloud provider reclaiming your VMs Learn more about Spot instances here Executors running out of memory Diagnosing Failing Jobs: Steps to Diagnose Failing Jobs Identifying Failing Jobs Click on the failing job to access its page Reviewing Failure Details Scroll down to see the failed stage Check the failure reason Diagnosing Failing Jobs: Generic Errors You may get a generic error. Click on the link in the description to see if you can get more info: Diagnosing Failing Jobs: Memory Issues Task Failure Explanation Scroll down the page to see why each task failed Memory issue identified as the cause Scenario – Spot instance , Auto-scaling Diagnosing Failing Executors: Checking Event Log Check Event Log Identify any explanations for executor failures Spot Instances Cloud provider may reclaim spot instances Diagnosing Failing Executors: Navigating to Executors Tab Check Event Log for Executor Loss Look for messages indicating cluster resizing or spot instance loss Navigate to Spark UI Click on the Executors tab Diagnosing Failing Executors: Getting Logs from Failed Executors Here you can get the logs from the failed executors: Scenario - Memory Issues Memory Issues Common cause of problems Requires thorough investigation Quality of Code Potential source of memory issues Needs to be checked for efficiency Data Quality Can affect memory usage Must be organized correctly Spark memory issues - Azure Databricks | Microsoft Learn Identifying Longest Stage Identify the longest stage of the job Scroll to the bottom of the job’s page Locate the list of stages Order the stages by duration Identifying Longest Stage Identify the longest stage of the job Scroll to the bottom of the job’s page Locate the list of stages Order the stages by duration Stage I/O Details High-Level Data Overview Input Output Shuffle Read Shuffle Write Number of Tasks in Long Stage Identifying the number of tasks Helps in pinpointing the issue Look at the specified location to determine the number of tasks Investigating Stage Details Investigate Further if Multiple Tasks Check if the stage has more than one task Click on the link in the stage’s description Get More Info About Longest Stage Click on the link provided Gather detailed information Conclusion Potential Data Skew Issues Data skew can impact performance May cause uneven distribution of data Spelling Errors in Data Incorrect spelling can affect data processing Ensure data accuracy for optimal performance Learn More Navigate to Skew and Spill - Skew and spill - Azure Databricks | Microsoft Learn
