A Practical Guide to Implement End-to-End Data Warehousing PoC Using Microsoft Fabric-Part1
Sometime back we had the opportunity to help a customer implement a PoC using Microsoft Fabric. In a series of blogs, we will demonstrate how we helped our customer from the SfMC team to implement a Data Warehouse PoC using the ELT (Extract, Load, Transform) approach. For this PoC, we used sample data from SQL Server’s well-known sample databases, giving us a straightforward and accessible way to illustrate the capabilities of Microsoft Fabric in a real-world data warehousing scenario. The following were the customer requirements: Build the enterprise grade Data Warehouse solution in Microsoft Fabric Ingest data from 59 diverse sources across 130 different countries Source data from ADLS (JSON) Source data from Azure and on-prem SQL Server databases Other data sources. PoC scope: ELT approach ingest (OLTP DB & ADLS Gen2) data directly into the Warehouse Transformation using stored procedures to load the production schema Solution Summary: Ingest OLTP data from source SQL Server (full load) Meta data driven copy activity to load data into staging schema of the Data Warehouse. Stored procedures to process the staging data along with data from ADLS shortcuts. Load the production Dimension tables. Application’s customer data arrives into ADLS storage as JSON files. Stored Procedures to process the data and load Customer dimension and Fact data using incremental load. Built 6 Dimensions and 1 Fact. Build reports. Sample overview of items that were implemented: Sample report from the PoC: Prerequisites: To implement this PoC, the following are the prerequisites: you may need to download the following files: You can download them from: Download all the required files to local drive (eg: C:\temp). We assume that you already have a Fabric tenant setup with a Fabric Capacity (you will need a F64 capacity to test the co-pilot feature else a F2 capacity will be sufficient). If Fabric is not enabled, use the link below to enable Fabric for your organization. Step by Step Guide to Enable Microsoft Fabric for Microsoft 365 Developer Account You will also need: A logical SQL Server (how to deploy one, how to add local IP address to the firewall exception) ADLS Gen2 Account (how to deploy one, how to create a container) Once created, Once the account is created, navigate to the “Data Storage” section and create a container name it levelup or a name of choice Open the container “levelup”and create a folder called “JSON_FILES” Install Storage Explorer, configure to connect to your Azure subscription, and navigate to storage account container. Upload five JSON files from the downloaded folder “JSON_FILES” to ADLS Gen2 in the "levelup" container under the "JSON_FILES" folder. Upload the following folders by choosing “Upload Folder” from Azure Storage Explorer. Sales_SalesOrderDetail Sales_SalesOrderHeader The above two folders containing Delta files, will be used for creating shortcuts in the Lakehouse which will be used for building the tables within the Warehouse in the subsequent tasks. After the upload, you should have the folders below inside your Levelup container. Next, create a database on the Azure SQL Server by using the bacpac files downloaded earlier: Connect to Azure SQL Logical Server using SSMS (if you don’t have, you can download here) Right click on Databases and select the option “Import Data-Tier Application”. Follow the screen captures below to complete the database bacpac import. Note: Depending upon DTU/V-core chosen, the Import activities might take up 30 mins. Continue to the next blogA Practical Guide to Implement End-to-End Data Warehousing PoC Using Microsoft Fabric-Part2
In this continuation (part 2) of the previous blog, you will create the required Fabric items within a workspace and create the tables in Lakehouse and Data Warehouse. Task1: Create a Microsoft Fabric Workspace, Data Warehouse and Lakehouse Before you can start building the solution, you'll first need to create a workspace where you'll create the rest of the items. Follow the steps below to create the workspace. Sign in to https://app.fabric.microsoft.com/ Select Workspaces > New Workspace. Fill out the Create a workspace dialog as follows: Name: LevelUpFabric1 (or a name of choice). Name should be unique. Expand the Advanced section. In the License Mode choose Trial or Fabric Capacity. Click Apply. The workspace will be created and opened. In the upper left corner, select New Item> search and click on Warehouse to create Data Warehouse and name it “DW_Levelup”. This will create the Data Warehouse and open it. Click on the workspace icon the left navigation to navigate to the workspace page. In the upper left corner of the Workspace, select New Item > search and click on Lakehouse to create a Lakehouse and name it, “LH_Levelup”. This will create the Lakehouse and open it. Task2: Create table Shortcuts in the Lakehouse from ADLS Gen2 We will now create shortcuts in the Lakehouse pointing to the two delta folders (Sales_SalesOrderHeader and Sales_SalesOrderDetail) which you had uploaded into the ADLS Gen2 store account in the pre-requisite steps. In the Lakehouse explorer, hover over the Tables folder (not Files folder) and click on the ellipsis and click “New Shortcut”. In the New Shortcut dialog box, select ADLS Gen2 under External sources. In the next screen, provide the required details as shown in the below snip. You may refer to your storage account settings for details. It is discouraged to use Account key for authentication but is done so for ease of use. The URL should be similar to: https://xxxxx.dfs.core.windows.net/levelup Ensure the shortcut name “Sales_SalesOrderDetail” and “Sales_SalesOrderHeader” matches the delta folder name. Important: Please use Sales_SalesOrderHeader and Sales_SalesOrderDetail as shortcut names. These names are used in the stored procedures. If you choose to use different shortcut names in step 5, you may need to update the SPs that refer to it, to avoid errors. The shortcuts should show up within the tables section of the Lakehouse like the below. We will use these shortcuts later in the solution. You may have to refresh the tables section for the shortcuts to appear (you may also have to refresh the tables section if the shortcuts appear as under “Unidentified” too. If it continues to be unidentified, there could be some steps above which were not followed correctly.) Task3: Create the required Tables and SPs in the Fabric Data Warehouse Now, go to Fabric workspace “LevelUpFabric1” and Navigate to the Data Warehouse and open a new query window and run below scripts. Note: The above scripts are available in the “FabricLevelUp_Files” folder that was downloaded during the prerequisite steps within “TSQL_Script” folder. Scripts Details 01_create_AllStgDimTbls.sql Creates staging (stg) schema and all the staging related tables. 02_TablesToCopy.sql Creates metadata tables to dynamic full load activity. 03_del_AllRecordsOfStgDimTbls.sql Creates stored procedure to delete all the records of the staging dim tables. 04_crt_Prd_Sch_Tbl_Sp.sql Create production (prd) schema, fact table, and SP for incremental update to fact table. 05_crt_Prd_Dim_Tbls.sql Create stored procedure to create all the production dim tables. 06_upd_prd_dim_Customer.sql Creates SP to update incremental stg.DimCustomer records to prd.DimCustomer tables. So far, we have created workspace, all the required items in the Lakehouse and Data Warehouse which will be used to load the Data Warehouse. Continue to the next blog...Create an Organizational Assets Library (including Multi-Geo & Information Barriers guidance)
Overview This guide walks through a practical approach to setting up SharePoint Online (SPO) Organizational Assets Libraries (OAL). It includes optional guidance for more complex tenants—such as Multi-Geo and Information Barriers (IB) - because those scenarios are often under-documented. What you’ll accomplish: Create and register Organizational Assets Libraries so templates, fonts, and brand images are available in Office apps, with notes for Multi-Geo, Information Barriers, Brand Center, and Copilot integration where applicable. Applies to: Standard (single-geo) tenants, Multi-Geo tenants, tenants with Information Barriers, and environments using Brand Center and/or Copilot features for organizational assets. Quick start (standard single-geo tenant) Create a SharePoint site to host Organizational Assets Libraries (often the Brand Center site). Create three document libraries (typical): ImageAssets, DocumentAssets (templates), FontAssets. Grant your intended audience Read access (commonly Everyone except external users via the site’s Visitors group). Enable the SharePoint Online Public CDN (tenant setting). Add a Public CDN origin for each library path (one origin per library). Upload approved assets (images, templates, fonts) into their respective libraries. Register each library with Add-SPOOrgAssetsLibrary (repeat per library). Validate registration and end-user experience, then allow up to 24 hours for Office apps to reflect changes. If you’re Multi-Geo or using Information Barriers: follow the same flow, but repeat per geo and complete registration while the site is in Open IB mode (details below). Key constraints and gotchas Multi-Geo: plan a repeatable per-geo pattern (typically one Org Assets site + matching libraries per geo) and keep naming consistent. Information Barriers (IB): Add-SPOOrgAssetsLibrary cannot be run when the target site is segmented—create and register libraries first (site in Open mode), then segment if needed. The “Everyone except external users” principal may be hidden by default, but it’s still commonly used for broad read access. Brand Center: many orgs host Org Assets Libraries in the Brand Center site; if Brand Center is created after libraries exist, it typically detects and uses them automatically. A public CDN must be enabled to support Organizational Assets Libraries. The “Everyone except external users” principal may be hidden by default, but it’s still commonly used for broad read access. Brand Center: many orgs host Org Assets Libraries in the Brand Center site; if Brand Center is created after libraries exist, it typically detects and uses them automatically. A public CDN must be enabled to support Organizational Assets Libraries. Implementation steps Prerequisites: SharePoint Online Management Shell access (or equivalent), permission to manage tenant settings, and the ability to create sites and libraries in each geo. Create a site to host your Organizational Assets Libraries (many orgs use a communication site). For ease of support, keep the site name, library names, and structure consistent over time. Note: A Communication site is recommended, but a Team site can also work. Example site URLs: In a standard tenant you’ll have one site; in Multi-Geo you’ll typically use one per geo. Primary geo: https://contoso.sharepoint.com/sites/BrandCenter EUR geo: https://contosoEUR.sharepoint.com/sites/BrandCenter APC geo: https://contosoAPC.sharepoint.com/sites/BrandCenter If your tenant uses Information Barriers, keep each site in Open IB mode while creating the Org Assets Libraries. You can segment the site later (if required) after libraries are created. Configure a public CDN (required) To use Brand Center and Organizational Assets Libraries, configure SharePoint Online to use a Public CDN. Set-SPOTenantCdnEnabled -CdnType Public -Enable $true Example output: Public CDN enabled locations: SITES/BRANDCENTER/FONTS */MASTERPAGE (configuration pending) */STYLE LIBRARY (configuration pending) */CLIENTSIDEASSETS (configuration pending) Note: You will see the new CDN is in a pending state until complete. This will take some time. Wait for the CDN to finish provisioning. Re-run the status/list commands until “pending” entries clear. Get-SPOTenantCdnEnabled -CdnType Public Get-SPOTenantCdnOrigins -CdnType Public Add CDN origins for each library Add allowed CDN origins for each asset library path (typically one origin per library). Example: Add-SPOTenantCdnOrigin -OriginUrl sites/BrandCenter/ImageAssets -CdnType Public Add-SPOTenantCdnOrigin -OriginUrl sites/BrandCenter/TemplateAssets -CdnType Public Add-SPOTenantCdnOrigin -OriginUrl sites/BrandCenter/FontAssets -CdnType Public Set permissions (required for broad consumption) To ensure most users can consume the assets, grant Everyone except external users (often abbreviated as EEEU) Read access (commonly via the site’s Visitors group). Example: add Everyone except external users to the Visitors group of the Organizational Assets site. Connect-SPOService -Url 'https://contoso-admin.sharepoint.com' $tenant = "9cfc42cb-51da-4055-87e9-b20a170b6ba3" $site = Get-SPOSite -Identity "https://contoso.sharepoint.com/sites/BrandCenter" $group = Get-SPOSiteGroup $site -Group "BrandCenter Visitors" Add-SPOUser -LoginName ("c:0-.f|rolemanager|spo-grid-all-users/" + $tenant) -Site $site -Group $group.Title Note: Organizational Assets Libraries respect SharePoint security trimming. If you need a narrower audience, grant Read to the appropriate groups instead of tenant-wide access. In many environments, Everyone except external users is required during registration (Add-SPOOrgAssetsLibrary) so Office can enumerate the library—test and confirm in your tenant before removing broad access. Create libraries and upload assets Create a document library for each asset type you plan to publish (for example: images, Office templates, fonts). For example: Upload your assets into the appropriate libraries. Example: Register each library using Add-SPOOrgAssetsLibrary. For this to work, Everyone except external users must already have access to the site (for example, via the Visitors group). Office Template Library Example: Add-SPOOrgAssetsLibrary -LibraryUrl 'https://contoso.sharepoint.com/sites/BrandCenter/DocumentAssets' -OrgAssetType OfficeTemplateLibrary Image Document Library Example: Add-SPOOrgAssetsLibrary -LibraryUrl 'https://contoso.sharepoint.com/sites/BrandCenter/ImageAssets' -OrgAssetType ImageDocumentLibrary Font Document Library Example: Add-SPOOrgAssetsLibrary -LibraryUrl 'https://contoso.sharepoint.com/sites/BrandCenter/FontAssets' -OrgAssetType OfficeFontLibrary -CdnType Public Optional: Enable Copilot support for an image library (only applicable to ImageDocumentLibrary). Set-SPOOrgAssetsLibrary -LibraryUrl 'https://contoso.sharepoint.com/sites/BrandCenter/ImageAssets' -OrgAssetType ImageDocumentLibrary -CopilotSearchable $true Multi-Geo mini runbook (recommended pattern) Use this as a simple tracking sheet so each geo ends up with a complete, consistent setup. Geo Site URL Libraries CDN origins added Libraries registered Primary https://<tenant>.sharepoint.com/sites/<BrandCenterOrAssetsSite> ImageAssets / DocumentAssets / FontAssets Yes/No Yes/No EUR https://<tenant>EUR.sharepoint.com/sites/<BrandCenterOrAssetsSite> ImageAssets / DocumentAssets / FontAssets Yes/No Yes/No APC https://<tenant>APC.sharepoint.com/sites/<BrandCenterOrAssetsSite> ImageAssets / DocumentAssets / FontAssets Yes/No Yes/No Naming standard (strongly recommended): keep the same site path and the same library names in every geo (for example, always ImageAssets, DocumentAssets, FontAssets). This minimizes per-geo scripting differences and reduces support effort. Wrap-up At this point, each geo should have its own site, libraries, CDN origins, and registered Organizational Assets Libraries. From here, focus on governance (who can publish/approve assets), naming standards, and ongoing lifecycle management (retire old templates/fonts and keep branding current). Validate configuration Admin checks (PowerShell) Confirm the Public CDN is enabled. Confirm CDN origins include one entry per assets library path. List registered Org Assets Libraries and verify each URL + type is present. Get-SPOTenantCdnEnabled -CdnType Public Get-SPOTenantCdnOrigins -CdnType Public Get-SPOOrgAssetsLibrary End-user checks (Office apps) In PowerPoint/Word, confirm organizational templates appear in the template picker (if you registered an OfficeTemplateLibrary). In Office font lists, confirm your org fonts appear (if you registered an OfficeFontLibrary). For image libraries, confirm approved brand images appear in supported pickers; if you enabled -CopilotSearchable, confirm images are discoverable as expected. Timing: New registrations and updates can take up to 24 hours to appear in Office apps. If you updated content, run Set-SPOOrgAssetsLibrary for each changed library, then wait for propagation. Updating content in existing Org Assets Libraries If you already have Organizational Assets Libraries registered and you need to publish updated templates, fonts, or images, use the process below. The high-level flow is: update content → run Set-SPOOrgAssetsLibrary (per library) → wait for propagation. Replace or update content in each library. Upload the new versions of templates/fonts/images into the appropriate library (and remove/retire older versions if needed). If Multi-Geo applies, repeat per geo. Update the matching libraries in each geo’s site so users in each geo get the same (or intentionally regional) set of assets. Run Set-SPOOrgAssetsLibrary for each updated library. Execute the cmdlet against the library URL to refresh the configuration after content changes (run it once per library you updated). Wait for Office app propagation. Allow up to 24 hours for updates to begin showing in Office apps. Example: Set-SPOOrgAssetsLibrary -LibraryUrl 'https://contoso.sharepoint.com/sites/BrandCenter/DocumentAssets' -OrgAssetType OfficeTemplateLibrary Notes: If your site is segmented by Information Barriers, confirm the cmdlet behavior in your environment before making changes, and prefer performing registration/updates while the site is in Open mode when possible. For image libraries, if you are using Copilot integration settings (for example -CopilotSearchable), keep the setting consistent when you run Set-SPOOrgAssetsLibrary. Make sure the intended audience still has Read access to the site/library; otherwise users may not see updates due to security trimming. Please note: After registering (or updating) your assets libraries, it can take up to 24 hours before changes become available in Office apps. Once fully enabled, Office apps will surface your templates and fonts. Below is an example. Example of interacting with Org Assets from M365 Apps Org Fonts from PowerPoint: From SharePoint: From Office Apps: Troubleshooting tips If Add-SPOOrgAssetsLibrary fails, confirm the site is not segmented by Information Barriers (Open mode during setup). If assets don’t appear in Office apps, wait for propagation (up to 24 hours) and re-check that the library was registered successfully. If CDN commands show “pending”, allow time for provisioning and re-run the status command. If users can’t see assets, verify the site/library permissions include Everyone except external users (or the intended audience group). Guidance: Using the SharePoint Online Public CDN Enabling the SharePoint Online Public CDN is a required and supported configuration for Organizational Assets Libraries, Brand Center, and related Office experiences. While the word “public” can sound concerning, it’s important to understand what is (and is not) exposed. We take great care to protect the data that runs your business. Data stored in the Microsoft 365 CDN is encrypted both in transit and at rest, and access to data in the Microsoft 365 SharePoint CDN is secured by Microsoft 365 user permissions and token authorization. Requests for data in the Microsoft 365 SharePoint CDN must be referred (redirected) from your Microsoft 365 tenant or an authorization token won't be generated. See: Content delivery networks - Microsoft 365 Enterprise | Microsoft Learn What “Public CDN” actually means Only explicitly approved library paths are cached The CDN does not expose your entire tenant. Administrators must explicitly register CDN origins (specific library paths). If a library is not registered as a CDN origin, it is not served via the CDN. No new content types are exposed The CDN is intended for static, non-sensitive assets such as: Brand images Office templates Fonts It is not designed for documents containing confidential or regulated data. Why Microsoft requires a Public CDN for Org Assets? Performance and reliability Office clients worldwide retrieve assets faster using geographically distributed edge caching. This avoids repeated downloads from SharePoint origin sites. Consistent Office app experiences PowerPoint, Word, Excel, and Copilot rely on CDN-backed delivery to surface: Templates Fonts Brand images Without a public CDN, these features may not function correctly or at all. Best practices Use the practices below to keep Organizational Assets Libraries reliable, secure, and easy for end users to adopt. Where relevant, notes call out additional considerations for Multi-Geo, Information Barriers, Brand Center, and Copilot. Governance and ownership checklist Owners/publishers: named group who can add/change assets (limited membership). Approvals: defined review/approval step before publishing new templates/fonts/images. Versioning/retention: how you retire old assets and prevent outdated branding from appearing in pickers. Rollback plan: how to revert a bad template/font/image quickly. Change communication: how you notify users about new/updated assets and expected timing (up to 24 hours). Assign clear owners (typically Brand/Comms) and a small admin group (typically IT) for each geo’s library and site. Decide what is “approved” vs “draft” content, and enforce it with a simple publishing process (for example, a review checklist or an approvals flow). Version and retire assets deliberately: keep one “current” template set and archive old assets to prevent users from picking outdated branding. Information architecture and naming Keep library names and structures consistent across geos (same library names, same folder conventions) to simplify support and documentation. Use descriptive filenames users can recognize in pickers (for example, “Contoso_Proposal_Template_v3”). Prefer a small number of clearly defined libraries by asset type (images, templates, fonts) rather than many small libraries. Permissions and access Ensure your intended audience has at least Read access to the site and libraries; Organizational Assets still follow SharePoint security trimming. If you use broad access (for example, Everyone except external users), document it and pair it with tight contributor permissions so only approved publishers can change assets. Avoid breaking inheritance in ways that make troubleshooting difficult—keep permissions simple and predictable whenever possible. CDN configuration Plan CDN changes ahead of time: enabling and provisioning can take time, and changes may not be immediate. Register only the origins you need (one per assets library path) and keep them consistent across environments. After changes, allow for propagation time before validating in Office apps. Multi-Geo and Brand Center Use a repeatable pattern: one site + matching libraries per geo, with the same structure and operational runbook. Be aware Brand Center is created in the primary geo; confirm how your org wants to manage global vs regional assets. Document which assets are global (shared everywhere) vs regional (geo-specific) to avoid confusion for publishers and users. Information Barriers (IB) sequencing Create and register Org Assets Libraries before segmenting the site when IB is enabled (create while the site is in Open mode, then segment later if required). After segmentation, re-validate that the right audience can still read the libraries (and that publishers can still manage content). Copilot readiness (image libraries) Use consistent, high-quality metadata for images (titles, descriptions, and tags). Copilot search quality depends heavily on this. If enabling image tagging integration, standardize on a tagging vocabulary (for example, brand terms, campaigns, departments, regions) so results are predictable. Only enable Copilot searchable settings on libraries where content is approved and intended for broad reuse. Q&A Q: What is an Organizational Assets Library (OAL)? A: It’s a SharePoint document library (or set of libraries) that you register so Office apps can surface approved templates, fonts, and images to users directly within the app experience. Q: Do I need SharePoint Brand Center to use OAL? A: No. You can use Organizational Assets Libraries without Brand Center. Brand Center can make asset management more accessible, for example, allowing SharePoint sites to use organizational branding, but OAL can be configured on its own. Q: Why is a “Public CDN” required, and is it safe? A: Office experiences rely on CDN-backed delivery for performance and reliability. “Public CDN” does not mean your whole tenant is exposed—only the specific library paths you register as CDN origins are cached. Access is still governed by Microsoft 365 authentication, token authorization, and SharePoint permissions. Q: Can I use this guide in a standard (single-geo) tenant? A: Yes. In a standard tenant you usually create one site and one set of libraries. The Multi-Geo guidance is only needed if your tenant is Multi-Geo (in which case you’ll typically repeat the pattern per geo). Q: How do Information Barriers (IB) affect setup? A: If a site is segmented, Add-SPOOrgAssetsLibrary cannot register the library. Create the site and register the libraries while the site is in Open mode, then segment afterward if required. Q: Why does “Everyone except external users” (EEEU) matter? A: In many environments, EEEU is required during library registration so Office can enumerate the library. However, OAL still respects SharePoint security trimming. If broad internal availability is the goal, a common pattern is to grant EEEU Read (often via the Visitors group) so Office apps can surface the assets to most internal users. If you need a narrower audience, use a group instead. Q: How long until assets show up (or update) in Office apps? A: It can take up to 24 hours for new registrations or updates to propagate. If you replaced content in an existing library, run Set-SPOOrgAssetsLibrary for each updated library, then allow time for Office apps to refresh. Q: How do I update content in an existing Org Assets Library? A: Replace the files in the library (and repeat across geos if applicable), then run Set-SPOOrgAssetsLibrary against each library you updated. After that, allow up to 24 hours for the updated assets to start showing in Office apps. Q: Do I need to run Set-SPOOrgAssetsLibrary every time I replace files? A: If you want Office apps to reliably pick up changes, run Set-SPOOrgAssetsLibrary after you update content (especially when publishing new/updated templates, fonts, or images). Treat it as the “refresh” step, then wait for propagation. Q: When should I enable Copilot support (CopilotSearchable) for an image library? A: Enable it only for libraries that contain approved, broadly reusable images and have strong metadata (title/description/tags). This helps ensure search results are on-brand and reduces the chance of surfacing unreviewed content. Q: Can I undo this later? A: Yes. You can unregister an Organizational Assets Library using SharePoint Online PowerShell (for example, Remove-SPOOrgAssetsLibrary) and remove CDN origins if you no longer need them. Plan governance so you can retire assets cleanly without disrupting users. Q: Users can’t see the assets (or updates)—what should I check first? A: Start with (1) permissions to the site/library (security trimming), (2) successful registration via Add-SPOOrgAssetsLibrary, (3) if you’re expecting an update, confirm you ran Set-SPOOrgAssetsLibrary for that library, (4) CDN provisioning status and configured origins, and (5) propagation time (up to 24 hours). Additional Reading Create an organization assets library - SharePoint in Microsoft 365 | Microsoft Learn Connect organizational asset libraries to Copilot for an on-brand experience - SharePoint in Microsoft 365 | Microsoft Learn Connect organizational asset libraries to PowerPoint for an on-brand experience - SharePoint in Microsoft 365 | Microsoft Learn Set up and connect organizational asset library (OAL) with image tagging to Copilot search | Microsoft Learn Add-SPOOrgAssetsLibrary (Microsoft.Online.SharePoint.PowerShell) | Microsoft Learn SharePoint Brand Center - SharePoint in Microsoft 365 | Microsoft Learn How to Enable Enterprise Brand Images with PowerPoint Copilot - SharePoint in Microsoft 365 | Microsoft Learn Office 365 Content Delivery Network (CDN) Quickstart - Microsoft 365 Enterprise | Microsoft Learn Use Office 365 Content Delivery Network (CDN) with SharePoint Online - Microsoft 365 Enterprise | Microsoft Learn Content delivery networks - Microsoft 365 Enterprise | Microsoft Learn Multi-Geo Capabilities in OneDrive and SharePoint - Microsoft 365 Enterprise | Microsoft Learn Use Information Barriers with SharePoint | Microsoft LearnA Deep Dive into Spark UI for Job Optimization
Key Insights for Spark Job Optimization The Spark UI is your X-ray into application execution: It provides real-time and post-mortem insights into every job, stage, task, and resource usage, moving you from guesswork to evidence-driven tuning. Systematic analysis is crucial: Start from high-level overviews in the Jobs tab, drill down into Stages for bottlenecks and shuffle operations, examine Tasks for skew and spills, and review Executors for resource allocation issues. Targeted optimizations yield significant gains: Address issues like data skew, excessive shuffles, memory pressure, and inefficient SQL plans with specific techniques such as repartitioning, broadcast joins, Kryo serialization, and proper resource allocation. Apache Spark is a powerful distributed computing framework, but extracting its full potential often requires meticulous optimization. The Spark UI (User Interface) stands as an indispensable tool, offering a detailed, web-based dashboard that provides real-time and historical insights into your Spark applications. It's the diagnostic center that helps you pinpoint performance bottlenecks, understand resource consumption, and identify inefficiencies that may be hindering your jobs. This comprehensive guide will walk you through the process of accessing, navigating, and interpreting the Spark UI, empowering you to translate its rich data into concrete strategies for optimizing your Spark jobs. As of July 1, 2025, modern Spark versions like 4.0.0 place significant emphasis on UI-driven performance tuning, making this a critical skill for any data professional. Accessing and Navigating the Spark UI: Your Diagnostic Gateway Before diving into optimization, you need to know how to access the Spark UI. Its accessibility varies depending on your Spark deployment mode: Local Mode: When running Spark locally, the UI is typically available at http://localhost:4040. Cluster Mode: In cluster environments like YARN, Mesos, or Kubernetes, the UI is usually accessed via the Spark History Server (often at port 18080) for post-mortem analysis, or through the application master's URL while the job is running. Cloud Platforms: On cloud services such as AWS Glue, Databricks, or EMR, the Spark UI is typically integrated into their respective consoles or accessible by enabling Spark event logging. Ensure event logs are configured to roll over to prevent metrics truncation for long-running jobs. Once accessed, the Spark UI is structured into several key tabs, each providing a different lens into your application's behavior: Jobs Tab: High-level overview of all jobs. Stages Tab: Detailed breakdown of stages within a job. Tasks Tab: Granular information about individual task execution. Storage Tab: Insights into cached RDDs and DataFrames. Environment Tab: Spark configuration and system properties. Executors Tab: Resource usage of individual executors. SQL Tab: Specific details for SQL queries and DataFrame operations (if applicable). Deciphering the Spark UI: A Tab-by-Tab Analysis An overview of the Jobs tab in the Apache Spark UI, showing job progress and details. 1. The Jobs Tab: Your Application's Pulse Check The Jobs tab is your initial point of contact for understanding the overall health and progress of your Spark application. It summarizes all submitted jobs, their status (running, completed, failed), duration, and general progress. This tab helps you quickly identify jobs that are stalling, taking excessively long, or have failed outright. What to look for: Overall Duration: Identify jobs that exhibit long durations. These are prime candidates for deeper optimization. Status and Progress: Observe jobs that are stuck or show a high number of failed tasks, indicating potential underlying issues that need immediate attention. Event Timeline: This visual representation of the application's lifecycle, including job execution and executor activity, can reveal patterns of resource contention or uneven parallel execution. 2. The Stages Tab: Unveiling Bottlenecks Stages are the backbone of a Spark job's execution, representing a sequence of tasks that can run together without data shuffling. The Stages tab provides granular details about each stage, making it crucial for pinpointing specific bottlenecks. The Stages tab in Spark UI, displaying detailed information for each stage of a job. Key Metrics and Analysis: Duration: Sort stages by duration to identify the longest-running ones. These are where your optimization efforts will likely yield the greatest impact. Input/Output (I/O) Sizes: High input/output metrics suggest that the stage might be I/O-bound. This points to opportunities for optimizing data formats or storage. Shuffle Read/Write: These are critical metrics. High "Shuffle Read" or "Shuffle Write" values indicate significant data movement between nodes, which is a very expensive operation. This often signals inefficient joins, aggregations, or partitioning. Task Progress and Event Timeline: Within the detail view of a stage, the event timeline visually represents individual task execution. Look for "straggler" tasks – tasks that take significantly longer than others – as this is a strong indicator of data skew where certain partitions hold disproportionately more data or require more computation. DAG Visualization: The Directed Acyclic Graph (DAG) visualization within a stage illustrates the flow of RDDs/DataFrames and the operations applied to them. This visual can simplify understanding complex data transformations and dependencies. For example, if a stage shows 3.2 TB of shuffle read and one task processes 400 GB compared to a median of 25 GB, this immediately highlights a severe data skew issue. 3. The Tasks Tab: Drilling Down to Individual Performance The Tasks tab offers the most granular view, showing execution details for individual tasks within a stage. This is where you can confirm observations from the Stages tab and identify specific issues like out-of-memory errors or high garbage collection times. Critical data points: Executor Run Time: Helps identify slow-running tasks. GC Time (Garbage Collection Time): High GC times indicate memory pressure and inefficient memory management, suggesting a need to optimize memory configurations or data serialization. Shuffle Spill (Memory Bytes Spilled / Disk Bytes Spilled): If tasks are spilling data to disk, it means they ran out of memory. This is a severe performance bottleneck, pointing to insufficient executor memory or inefficient data processing. Host: Sorting the task table by host can reveal skewed executors, where one executor is burdened with significantly more work due to data imbalance. 4. The SQL Tab: Optimizing Your DataFrames and SQL Queries For Spark DataFrame and SQL workloads, the SQL tab is invaluable. It provides detailed information about executed SQL queries, including their duration, associated jobs, and, most importantly, their physical and logical execution plans. Analyzing SQL queries: Physical Plan: This is a textual and graphical representation of how the Spark optimizer decided to execute your query. Look for inefficient join strategies (e.g., unintended Cartesian joins, inefficient Sort-Merge Joins where Broadcast Join would be better), missed filter pushdowns, or unnecessary data shuffles (indicated by "Exchange" operations). Graphical Visualization: This visual simplifies the analysis by showing aggregated information about rows and data processed at each stage of the SQL query. By analyzing the physical plan, you can validate whether your DataFrame transformations or SQL queries are being optimized as expected. For instance, if you've hinted for a broadcast join but the plan shows a Sort-Merge Join with a huge shuffle, you know there's a problem. 5. The Executors Tab: Resource Utilization Deep Dive This tab provides a detailed view of the resources consumed by each executor in your cluster, including CPU cores, allocated memory, used memory, disk usage, and the number of active tasks. It's essential for understanding resource allocation and identifying bottlenecks related to cluster configuration. Key checks: Memory Used vs. Total Memory: Identify if executors are underutilized or overloaded. High memory usage combined with disk spills indicates memory pressure. CPU Cores: Verify if your allocated CPU cores are being efficiently utilized. Low utilization might suggest insufficient parallelism or tasks waiting for resources. Disk Usage: Indicates if tasks are performing large I/O operations or spilling excessive data to disk. Thread Dump: Allows you to inspect the JVM thread dump on each executor for advanced debugging of performance issues. 6. The Storage Tab: Managing Cached Data If your Spark application uses caching or persistence (e.g., via cache() or persist()), the Storage tab provides details about persisted RDDs and DataFrames, including their storage levels (memory, disk, or both), sizes, and partition distribution. Insights from the Storage tab: Memory Management: Ensure cached data is not consuming excessive memory or being spilled to disk unnecessarily. Appropriate Caching Strategy: Verify that frequently accessed datasets are cached with suitable storage levels to minimize recomputation without causing memory overflows. 7. The Environment Tab: Configuration Validation This tab displays all Spark configuration properties, JVM settings, and system environment variables. It's a crucial place to confirm that your Spark application is running with the intended configurations. Key usage: Configuration Validation: Double-check if critical Spark configurations like spark.executor.memory, spark.executor.cores, spark.sql.shuffle.partitions, and spark.serializer are set correctly. Misconfigurations can severely impact performance. Translating UI Insights into Optimization Strategies Once you've analyzed the Spark UI and identified specific bottlenecks, you can apply targeted optimization techniques. This shift from "guess-and-check" to "evidence-driven" tuning can significantly improve job runtimes and reduce costs. 1. Addressing Data Skew Detection: Long "straggler" tasks in the Stage Event Timeline, uneven partition sizes, or highly skewed "Shuffle Read/Write" metrics in the Stages tab. Optimization: Repartitioning: Use repartition(N) or repartitionByRange(N, column) to distribute data more evenly across partitions. For instance, df = df.repartitionByRange(800, "customer_id") for a skewed customer_id key. Salting: For highly skewed join keys, add a random prefix (salt) to the key before joining, then remove it afterward. Adaptive Query Execution (AQE): In Spark 3.2+, enable AQE (spark.sql.adaptive.enabled=true and spark.sql.adaptive.skewJoin.enabled=true). AQE can dynamically detect and mitigate data skew during shuffle operations. 2. Optimizing Shuffles Detection: High "Shuffle Read" and "Shuffle Write" metrics in the Stages tab, indicating excessive data movement. Optimization: Filter Early: Push down filters and projections as early as possible to reduce the amount of data processed and shuffled. Broadcast Joins: For small tables (typically under spark.sql.autoBroadcastJoinThreshold, default 10MB), use broadcast(df) hint or set spark.sql.autoBroadcastJoinThreshold to enable broadcast joins. This avoids a shuffle for the smaller table. Adjust Shuffle Partitions: Configure spark.sql.shuffle.partitions appropriately. A common rule of thumb is 2-4 times the number of total executor cores, ensuring each partition is between 100-200 MB to avoid OOM errors and small file overhead. Coalesce: Use coalesce() for reducing the number of partitions without triggering a full shuffle if data size allows. 3. Memory Management and Garbage Collection Detection: High "Shuffle Spill" (Memory/Disk Bytes Spilled) in the Tasks tab, out-of-memory errors, or significant "GC Time" in the Executors tab or Task details. Optimization: Executor Memory: Increase spark.executor.memory if tasks are spilling to disk. Memory Fractions: Adjust spark.memory.fraction and spark.memory.storageFraction to allocate more memory for execution or caching. Serialization: Use Kryo serialization (spark.serializer=org.apache.spark.serializer.KryoSerializer) for faster and more compact data serialization, reducing memory footprint and network I/O. Caching: Cache only necessary DataFrames that are reused multiple times, and use appropriate storage levels (e.g., MEMORY_AND_DISK). Unpersist data promptly when no longer needed. GC Tuning: For large heaps, consider tuning JVM garbage collector settings, often involving the G1GC algorithm, to minimize GC pauses. High GC time (e.g., >15% of task time) indicates too many small objects. 4. Resource Allocation and Parallelism Detection: Underutilized executors (low CPU usage, many idle cores), tasks waiting for resources in the Jobs/Executors tabs, or dynamic allocation adding/removing executors frequently. Optimization: Executor Cores/Memory: Adjust spark.executor.cores and spark.executor.memory to match your cluster's capacity and workload. Ensure you have enough executors to handle the desired parallelism. Default Parallelism: Set spark.default.parallelism to a value that provides sufficient concurrent tasks, ideally 2-4 times the total number of CPU cores in your cluster. 5. SQL Query and DataFrame Optimization Detection: Inefficient physical plans in the SQL tab, long-running SQL queries, or unnecessary "Exchange" operations. Optimization: Predicate Pushdown: Ensure filters are applied as early as possible (e.g., directly in the data source read) to reduce the amount of data processed. Join Order and Strategy: Reorder joins to place selective filters and smaller tables first. Leverage specific join hints (BROADCAST, SHUFFLE_HASH) where appropriate. Column Pruning: Select only the columns you need, avoiding full table scans. Bucketing and Partitioning: For frequently joined or filtered columns, consider bucketing and partitioning your data to improve performance of joins and aggregations. This bar chart quantifies the common performance bottlenecks in Spark, indicating their typical impact on job execution on a scale of 0 to 10. Higher scores suggest more significant performance degradation. Understanding these high-impact areas helps prioritize optimization efforts. A Practical Example: Tackling Data Skew with the UI Imagine a PySpark ETL job that takes 48 minutes to complete. A quick glance at the Jobs tab shows that "Job 3" accounts for 42 of those minutes. Drilling into Job 3, the Stages tab reveals that "Stage 19" is the culprit, consuming 38 minutes and involving 3.2 TB of shuffle read. Further inspection of Stage 19's Event Timeline within the Stage Detail view immediately highlights a "straggler" task on a specific host (e.g., ip-10-0-4-11). This task processed an anomalous 400 GB of data, compared to the median 25 GB for other tasks in the same stage. This disparity is a classic symptom of data skew, likely caused by a highly skewed key like "customer_id". The Fix: Based on this evidence, an optimization is implemented: df = df.repartitionByRange(800, "customer_id") potentially combined with salting if the skew is severe. After redeploying, the Spark UI confirms the success: Stage 19's runtime drops to 6 minutes, the total job to 12 minutes, and crucially, there's no disk spill and GC time is less than 3%. This example underscores how the Spark UI provides the exact evidence needed to diagnose issues and validate the effectiveness of applied optimizations. Optimizing for the Future: Best Practices and Continuous Improvement Effective use of the Spark UI isn't a one-time activity; it's an ongoing process for continuous optimization. Table of Common Symptoms and Proven Fixes Symptom in UI Root Cause What to Change / Fix Few very long tasks; wide idle band at end of stage (stragglers) Too few partitions or severe data skew repartition(N) or repartitionByRange; for skew: salting, skew join hint, enable AQE skew mitigation Shuffle spill: "Disk Bytes Spilled" > 0 Executor memory insufficient Raise spark.executor.memory / spark.memory.fraction, use Kryo serialization, filter earlier Stage uses SortMergeJoin with huge shuffle where BroadcastJoin was expected Broadcast join not chosen or threshold too low broadcast(df) hint or configure spark.sql.autoBroadcastJoinThreshold GC Time > 15% of Task Time Too many small objects, inefficient memory usage cache() only necessary data, use Dataset encoders or vectorized Parquet reader, increase executor heap but watch GC algorithm Executors idle in timeline; dynamic allocation frequently adds/removes Slots > parallelism; poor partitioning for workload Lower spark.sql.shuffle.partitions, coalesce downstream if appropriate, adjust spark.default.parallelism SQL plan shows multiple "Exchanges" stacking Unnecessary repartitions (e.g., narrow-wide-narrow pattern) Use colocated sort-merge join hints, reuse partitioning columns, analyze query logic for redundant shuffles High I/O metrics in Stages tab (e.g., large input size without sufficient processing) Inefficient data format, full table scans, or lack of predicate pushdown Optimize data formats (e.g., Parquet with snappy compression), apply filters/projections early, leverage partitioning/bucketing in source data Application fails with OutOfMemoryError (OOM) on driver or executor Insufficient driver/executor memory for data or operations Increase spark.driver.memory or spark.executor.memory; reduce partition size or number of partitions; enable off-heap memory if applicable This table summarizes common symptoms observed in the Spark UI, their root causes, and corresponding solutions. It serves as a quick reference guide for targeted optimization efforts. Visualization of Spark UI Concepts This Mermaid mindmap visually organizes the key concepts related to analyzing the Spark UI and optimizing Spark jobs, covering accessing the UI, understanding its various tabs, specific optimization strategies, and overarching best practices for continuous improvement. Conclusion Analyzing the Spark UI is an art and a science, offering an unparalleled view into the inner workings of your Spark applications. By systematically navigating its various tabs—Jobs, Stages, Tasks, SQL, Executors, Storage, and Environment—you can gather crucial evidence to diagnose performance issues such as data skew, excessive shuffles, memory pressure, and inefficient resource allocation. This evidence-driven approach allows you to implement targeted optimizations, whether it's through repartitioning data, adjusting memory configurations, fine-tuning SQL queries, or optimizing resource allocation. Mastering the Spark UI not only transforms you into a more effective Spark developer but also ensures that your big data pipelines run with optimal efficiency, leading to significant reductions in execution time and operational costs. Continuous monitoring and iterative optimization based on UI insights are the keys to maintaining robust and performant Spark applications in production environments. Frequently Asked Questions (FAQ) What is the primary purpose of the Spark UI? The Spark UI serves as a web-based interface for monitoring, debugging, and optimizing Spark applications by providing real-time and historical insights into job execution, resource utilization, and performance bottlenecks. How can I access the Spark UI in a cluster environment? In a cluster environment, the Spark UI can typically be accessed via the Spark History Server (often running on port 18080) for completed jobs, or through the application master's URL while the job is still active. Cloud platforms like AWS Glue or Databricks usually provide direct links in their respective consoles. What does "Shuffle Read/Write" indicate in the Spark UI? "Shuffle Read/Write" metrics in the Stages tab indicate the amount of data transferred between executors across the network during shuffle operations. High values often point to expensive data redistribution, which can be a significant performance bottleneck, typically caused by wide transformations like joins or aggregations. How do "straggler" tasks relate to data skew? "Straggler" tasks are individual tasks within a stage that take significantly longer to complete than others. They are a primary indicator of data skew, where certain data partitions have disproportionately more data or require more computation, leading to uneven work distribution across executors. What are some immediate actions to take if the Spark UI shows high "Shuffle Spill"? High "Shuffle Spill" (data written to disk due to memory limitations) suggests that executors are running out of memory. Immediate actions include increasing spark.executor.memory, optimizing data serialization (e.g., using Kryo), or filtering data earlier to reduce memory footprint. Referenced Sources Performance Tuning - Spark 4.0.0 Documentation - spark.apache.org Diagnose cost and performance issues using the Spark UI - Databricks Documentation Web UI - Spark 4.0.0 Documentation - spark.apache.org Diagnose cost and performance issues using the Spark UI | Databricks Documentation How to interpret Spark UI - Databricks Community - 109593 Apache Spark Performance Tuning: 7 Optimization Tips (2025) Diagnose cost and performance issues using the Spark UI - Azure Databricks | Microsoft Learn Mastering Spark UI Monitoring in PySpark: Optimizing Performance ... Diagnose cost and performance issues using the Spark UI r/dataengineering on Reddit: Beginner’s Guide to Spark UI: How to Monitor and Analyze Spark Jobs Diagnose cost and performance issues using the Spark UI How to Optimize Spark Jobs for Maximum Performance Monitoring and Instrumentation - Spark 4.0.0 Documentation Spark Web UI - Understanding Spark Execution - Spark By {Examples} How to read Spark UI - Stack Overflow
Diagnose performance issues in Spark jobs through Spark UI.
Agenda Introduction Overview of Spark UI Navigating to Spark UI Jobs Timeline Opening Jobs Timeline Reading Event Timeline Failing Jobs or Executors Diagnosing Failing Jobs Diagnosing Failing Executors Scenario - Memory Issues Scenario - Long Running Jobs Scenario - Identifying Longest Stage Introduction Diagnosing performance issues of job using Spark UI This guide walks you through how to use the Spark UI to diagnose performance issues Overview of Spark UI Job Composition Composed of multiple stages Stages may contain more than one task Task Breakdown Tasks are broken into executors Navigating to Spark UI: Navigating to Cluster's Page Navigate to your cluster’s page: Navigating to Spark UI: Clicking Spark UI Click Spark UI: Jobs Timeline Jobs timeline The jobs timeline is a great starting point for understanding your pipeline or query. It gives you an overview of what was running, how long each step took, and if there were any failures along the way Opening Jobs Timeline Accessing the Jobs Timeline Navigate to the Spark UI Click on the Jobs tab Viewing the Event Timeline Click on Event Timeline Highlighted in red in the screenshot Example Timeline Shows driver and executor 0 being added Failing Jobs or Executors: Example of Failed Job Failed Job Example Indicated by a red status Shown in the event timeline Removed Executors Also indicated by a red status Shown in the event timeline Failing Jobs or Executors: Common Reasons for Executors Being Removed Autoscaling Expected behavior, not an error See Enable autoscaling for more details Compute configuration reference - Azure Databricks | Microsoft Learn Spot instance losses Cloud provider reclaiming your VMs Learn more about Spot instances here Executors running out of memory Diagnosing Failing Jobs: Steps to Diagnose Failing Jobs Identifying Failing Jobs Click on the failing job to access its page Reviewing Failure Details Scroll down to see the failed stage Check the failure reason Diagnosing Failing Jobs: Generic Errors You may get a generic error. Click on the link in the description to see if you can get more info: Diagnosing Failing Jobs: Memory Issues Task Failure Explanation Scroll down the page to see why each task failed Memory issue identified as the cause Scenario – Spot instance , Auto-scaling Diagnosing Failing Executors: Checking Event Log Check Event Log Identify any explanations for executor failures Spot Instances Cloud provider may reclaim spot instances Diagnosing Failing Executors: Navigating to Executors Tab Check Event Log for Executor Loss Look for messages indicating cluster resizing or spot instance loss Navigate to Spark UI Click on the Executors tab Diagnosing Failing Executors: Getting Logs from Failed Executors Here you can get the logs from the failed executors: Scenario - Memory Issues Memory Issues Common cause of problems Requires thorough investigation Quality of Code Potential source of memory issues Needs to be checked for efficiency Data Quality Can affect memory usage Must be organized correctly Spark memory issues - Azure Databricks | Microsoft Learn Identifying Longest Stage Identify the longest stage of the job Scroll to the bottom of the job’s page Locate the list of stages Order the stages by duration Identifying Longest Stage Identify the longest stage of the job Scroll to the bottom of the job’s page Locate the list of stages Order the stages by duration Stage I/O Details High-Level Data Overview Input Output Shuffle Read Shuffle Write Number of Tasks in Long Stage Identifying the number of tasks Helps in pinpointing the issue Look at the specified location to determine the number of tasks Investigating Stage Details Investigate Further if Multiple Tasks Check if the stage has more than one task Click on the link in the stage’s description Get More Info About Longest Stage Click on the link provided Gather detailed information Conclusion Potential Data Skew Issues Data skew can impact performance May cause uneven distribution of data Spelling Errors in Data Incorrect spelling can affect data processing Ensure data accuracy for optimal performance Learn More Navigate to Skew and Spill - Skew and spill - Azure Databricks | Microsoft LearnReducing SQL Connection Latency for Apps Using Azure AAD Authentication
Challenge: connection latency and token overhead Consider a cloud-native application deployed in Azure App Service or Kubernetes (AKS) that needs to query an Azure SQL Database for real-time data. The application uses Azure Active Directory (AAD) for secure authentication, but every time the application establishes a new connection to the database, it requests a new AAD token. In high-traffic environments where thousands of requests are processed per second, this repetitive token issuance introduces latency and performance degradation. This delay becomes particularly problematic for time-sensitive applications where every millisecond counts. Each token request impacts response times and creates unnecessary resource consumption. Solution: token caching and expiration management To mitigate these delays, we can optimize the authentication process by caching the AAD token and reusing it for the duration of its validity (typically 1 hour to 24 hours). Instead of requesting a new token for every database connection, the token is fetched only when the existing one is near expiration. This approach eliminates the repeated authentication overhead and ensures that the application can maintain seamless connectivity to the database without the performance hit of generating a new token for each request. In addition to reducing latency, this approach reduces the number of HTTP calls made to the Azure Active Directory service, resulting in better resource utilization and lower operational costs. Concrete performance gains: optimized SQL client connection As part of the mitigation, we provide a custom code implementation that uses SqlClient, a supported library, to optimize the connection time. The test was conducted with the S0 database, where using a single process and using connection pooling, we opened a connection, executed the SELECT 1, and closed the connection. During the testing phase with a connection pooler script running for 96 hours (without the AAD token cache), the following results were observed: 10 connections took 1 second, representing 0.866% of total connections. 1 connection took 4 seconds, representing 0.0866%. 1.144 connections took less than 1 second, representing 99.05% of total connections. All executions of SELECT 1 were completed in 0 seconds. These results demonstrate how caching AAD tokens and reusing them effectively reduced connection overhead and improved performance. None of the connections exceeded 5 seconds in duration, while with the default behavior, connections were reaching 30 seconds and more, depending on the environment complexity. Step-by-step implementation Here’s a step-by-step guide on how to implement this solution using C# and the Microsoft.Data.SqlClient package to optimize SQL database connections: Obtain and cache a token: Instead of requesting a new AAD token with every connection, we obtain a token once and cache it. This is done by leveraging Azure Managed Identity to authenticate the application, which eliminates the need to repeatedly authenticate with Azure Active Directory for every database connection. In this step, we fetch the token once and store it securely for reuse. Renew the token only when it’s near expiry We will refresh the token only when it is nearing expiration or has already expired. The application checks the token’s expiration time before attempting to use it. If the token is still valid, it continues to be reused. If it's close to expiration, a new token is fetched. Reuse a single token across multiple connections: The cached token can be used for multiple database connections during its lifetime. Rather than requesting a new token for each new connection, the application will use the same token across all connections until the token is about to expire. Code example: optimized SQL connection management Here’s an example of how you can implement token caching in a C# application using Microsoft.Data.SqlClient. using System; using System.Data.SqlClient; using System.Diagnostics; using System.Threading; using Azure.Identity; namespace SqlConnectionOptimization { public class SqlConnectionManager { private string _accessToken; private DateTimeOffset _tokenExpiration; private readonly string _connectionString = "Server=tcp:servername.database.windows.net,1433;Initial Catalog=DBName;..."; private readonly Stopwatch _stopwatch = new Stopwatch(); public SqlConnectionManager() { _accessToken = string.Empty; _tokenExpiration = DateTimeOffset.UtcNow; } public void Run() { while (true) { // Refresh token if necessary if (IsTokenExpired()) { RefreshToken(); } // Establish connection and perform operations using (var connection = CreateConnection()) { LogExecutionTime("Connected"); ExecuteQuery(connection); LogExecutionTime("Query Executed"); } // Simulate some idle time between operations Log("Waiting before next operation..."); Thread.Sleep(1000); } } private bool IsTokenExpired() { return string.IsNullOrEmpty(_accessToken) || DateTimeOffset.UtcNow.AddMinutes(5) >= _tokenExpiration; } private void RefreshToken() { _stopwatch.Start(); try { var result = FetchAccessToken(); _accessToken = result.Token; _tokenExpiration = result.Expiration; LogExecutionTime("Token Refreshed"); Log($"Token expires at: {_tokenExpiration}"); } catch (Exception ex) { Log($"Error fetching token: {ex.Message}"); } } private (string Token, DateTimeOffset Expiration) FetchAccessToken() { var managedIdentityCredential = new ManagedIdentityCredential(); var tokenRequestContext = new Azure.Core.TokenRequestContext(new[] { "https://database.windows.net/" }); var accessToken = managedIdentityCredential.GetTokenAsync(tokenRequestContext).Result; return (accessToken.Token, accessToken.ExpiresOn.UtcDateTime); } private SqlConnection CreateConnection() { var connection = new SqlConnection(_connectionString) { AccessToken = _accessToken }; int retries = 0; while (true) { try { connection.Open(); return connection; } catch (Exception ex) { retries++; if (retries > 5) { Log($"Error connecting after multiple retries: {ex.Message}"); throw; } Log($"Connection attempt failed. Retrying in {retries} seconds..."); Thread.Sleep(retries * 1000); } } } private void ExecuteQuery(SqlConnection connection) { var query = "SELECT 1"; // Simple query, replace with real logic as needed int retries = 0; while (true) { try { using (var command = new SqlCommand(query, connection)) { command.CommandTimeout = 5; // Adjust timeout for more complex queries command.ExecuteScalar(); } return; } catch (Exception ex) { retries++; if (retries > 5) { Log($"Max retries reached for query execution: {ex.Message}"); throw; } Log($"Query execution failed. Retrying in {retries} seconds..."); Thread.Sleep(retries * 1000); } } } private void Log(string message) { Console.WriteLine($"{DateTime.Now:yyyy-MM-dd HH:mm:ss.fff}: {message}"); } private void LogExecutionTime(string action) { _stopwatch.Stop(); var elapsed = _stopwatch.Elapsed; Log($"{action} - Elapsed time: {elapsed:hh\\:mm\\:ss\\.fff}"); _stopwatch.Reset(); } public static void Main(string[] args) { var manager = new SqlConnectionManager(); manager.Run(); } } } Key points in the code Token Expiration Check: The IsTokenExpired() method checks whether the token has expired by comparing it to the current time. We’ve added a 5-minute buffer for token expiration. This can be adjusted based on your needs. Managed Identity Authentication: The application uses Azure Managed Identity to authenticate and fetch the token, ensuring secure and scalable access to Azure SQL Database without requiring client secrets. Retry Logic: In the event of a connection failure or query execution failure, the system retries a set number of times with exponential backoff, making it resilient to transient network or authentication issues. Conclusion By implementing a token caching and expiration management strategy, applications can dramatically improve the performance and scalability of their database interactions, especially in environments with high request volumes. By leveraging Azure Managed Identity for secure, reusable tokens, you can reduce authentication latency and improve the overall efficiency of your SQL database connections. This approach can also be adapted to any service using Azure SQL Database and Azure Active Directory for authentication. Next steps Benchmarking: Test the implementation in your environment to quantify the performance gains. Error Handling: Extend the retry logic and error handling to better handle transient failures, especially in production environments. Resources: Introducing Configurable Retry Logic in Microsoft.Data.SqlClient v3.0.0-Preview1 Configurable retry logic in SqlClient Troubleshoot transient connection errors Scaling: Consider how this strategy can be applied across multiple services in larger architectures. Consider reading and applying managed identity best practices. Resources: Managed identity best practice recommendationsScaling PostgreSQL Connections in Azure: A Deep Dive into Multi-PgBouncer Architectures
This article dives into building a robust and scalable connection pooling layer for Azure Database for PostgreSQL - Flexible Server. You'll learn how to deploy multiple PgBouncer instances, managed by an Azure Load Balancer, to eliminate single points of failure and dramatically boost throughput for your applications.A Practical Guide to Implement End-to-End Data Warehousing PoC Using Microsoft Fabric-Part 3
In this continuation (part 3) of the previous blog, you will create a simple metadata driven pipeline to ingest data into the Data Warehouse and populate the dimension tables. Task4: Create a metadata driven pipeline to ingest data into the Data Warehouse and populate the dimension tables. We will use the table mtd.TablesToCopy created in Task3 of above while building a metadata driven pipeline, to ingest data from the Azure SQL Database. Click on the Workspace icon on the left side navigation and click “+ New Item” > search “Data pipeline” to create a new pipeline and provide a name (eg., PL_Dim_FullLoad_Levelup) Once the pipeline editor opens, click on the Activities tab and add a Stored Procedure activity and name it DeleteAllRowsfromStgTbls in the General tab On the settings tab, use the below snip to populate the details. This stored procedure deletes all the records of the staging dimension tables to facilitate a full load (except stg.DimCustomer) Note: We'll utilize stg.DimCustomer for incremental changes, won't be deleting all rows from the table. Click on the whitespace in the pipeline editor/canvas to add a parameter called “LookupTable” like the below. Add a lookup activity into the canvas. Connect the “On success” port of the “Stored Procedure” activity to the Lookup activity. From the general tab of the lookup activity, rename it to TablesToCopyLookup. In the settings tab, use the below snip to populate the details. Important: Remember to uncheck the First row only Note: In the above snip the Query field is set dynamically. To do so, click on the Add dynamic content as shown below and paste the following code: SELECT * FROM @{pipeline().parameters.LookupTable} The lookup activity will get the table name passed by the pipeline parameter at runtime and run the select statement against the table – mtd.TablesToCopy. Add a ForEach activity to the canvas and connect the “On success” port of the “Lookup” activity to the “ForEach” activity. The ForEach activity will iterate once for each value returned by the Lookup activity. Rename the activity name to ForEach_Row_In_TablesToCopy Set the settings for the ForEach activity as below. The “items” field should be set similar to note section of step 5. Paste the following into the items field - @activity('TablesToCopyLookup').output.value. Double click or edit the ForEach activity and within the ForEach activity, add a Copy Activity, set its name as “Copy_Data_From_SQL_To_Fabric_DW” in the General tab to: Set the source tab settings based on the snip below. In the connection field, setup a new connection to the Azure SQL DB that you created in the pre-requisites section. In the Query field click Add dynamic content and type - SELECT * FROM @{item().tableName}. This will pass the name of the table based on the iteration we are in. Set the destination tab settings based on the snip below. Notice that we will copy the source tables from the Azure SQL database to the staging schema and dynamically pass table names based on the source tables. The schema field is statically set to stg. To create the table names in the data warehouse dynamically, in the table name field, click Add dynamic content and paste - @concat('Dim',split(item().tableName,'.')[1]). This expression, concatenates the string “Dim” to the second part of the two part source table. In the canvas showing the Copy activity, click on Main canvas on the top left corner to exit out of the ForEach activity. Add a Stored procedure activity to the canvas and connect the Success port of the ForEach activity to the stored procedure. Rename the stored procedure activity from the General tab to - CTAS_For_PrdDimTbls. In the settings tab, fill out the details using the snip below Click on the Home tab > Save to save the pipeline and the Play button to execute the pipeline. The pipeline should look like the below. After a successful run, it would have copied all the tables from the Azure SQL Database to the stg schema of the Data Warehouse and would have populated all the Dimensions tables in the production schema.Azure Databricks - Best Practices for Using Spot Instances in Cluster Scaling
Leveraging Spot instances in Azure Databricks can greatly reduce costs; however, we strongly advise against their use for critical production workloads requiring high SLAs. Since Spot instances are subject to availability and can be interrupted at any time, they pose a risk to workload stability. If you still choose to use Spot instances for such workloads, it is essential to follow best practices to mitigate potential risks. Spot instances provide a cost-efficient way to scale compute clusters, but improper configurations may lead to instability and job failures. This blog outlines key best practices for using Spot instances, whether auto-scaling is enabled or disabled. When Auto Scaling is Disabled with Spot Instances Without auto-scaling, Spot instance availability is crucial for successful cluster startup. Here’s what you need to consider: Cluster Availability Ensure that 80% of the total requested nodes are available for startup. For instance, if you request four Spot compute worker nodes, the eviction of even a single node can delay the cluster's launch. Cluster Launch Attributes Use attributes like FALL_BACK_TO_AZURE during cluster launch. This ensures that if Spot instances are unavailable, an on-demand compute node will be provisioned instead, preventing cluster failures. Avoid Using Pools with Spot Instances Creating clusters from pools with Spot instances can introduce instability, especially if the driver node is assigned to a Spot instance. To prevent this, we strongly discourage using pools with Spot instances when launching clusters. When Auto Scaling is Enabled with Spot Instances Auto-scaling allows clusters to dynamically adjust resources, but careful setup is necessary for smooth scaling. On-Demand Nodes First Set the Azure attribute first_on_demand=2 in the job cluster definition. This ensures that the first two nodes (one driver and one worker) are on-demand, stabilizing cluster creation. Autoscaling Settings Enable auto-scaling on the cluster. Set min_workers=1, ensuring that at least one worker is always on-demand. Define the maximum cluster size to prevent over-scaling issues. This setup ensures reliable cluster startup and reduces the risk of job failures. Upscaling Considerations The cluster should always start with on-demand nodes before scaling up with Spot instances. While this approach improves stability, it may slightly increase overall job duration due to the upscaling process. Avoid Using Pools with Spot Instances Just like in the non-auto-scaling setup, avoid creating clusters from pools with Spot instances. Doing so can lead to delayed startups and instability. Final Thoughts By following these best practices, you can maximize the benefits of Spot instances while ensuring cluster stability and efficiency. Whether auto-scaling is enabled or not, prioritizing on-demand instances during startup and carefully managing scaling policies will help mitigate potential risks.
Your on-call rotation has a new member: 10 production incidents, end to end, with Azure SRE Agent
What this post is. A hands-on, reproducible walkthrough of ten real production failure modes — App Service, AKS, Azure SQL, Cosmos DB, VMs, VM Scale Sets, Application Gateway, and Service Bus — each one driven end to end by Azure SRE Agent: detection, hypothesis-driven investigation, ITSM ticketing, bounded remediation, recovery validation, and follow-up records. What this post is not. A claim that you install SRE Agent and all of this happens on day one. Every workflow below needs telemetry, scoped RBAC, a response plan, an approved action surface, and an ITSM integration. I'll be explicit about which parts are documented product behavior and which parts you have to wire yourself — because that distinction is the difference between a demo that works on stage and one that works at 3 AM. TL;DR The pattern Alert → agent investigates read-only → agent opens/updates the ticket with evidence → agent proposes a bounded action → human approves → agent executes → agent validates recovery → agent files the follow-up record The unlock Not "AI fixes prod." The unlock is that investigation — the 20 minutes of tab-switching between Azure Monitor, App Insights, deployment history, and Activity Log — is fully automated and consistent, and the fix arrives pre-justified with an evidence chain The guardrail Read-only automatic. Ticketing automatic. Production writes in Review mode. Guest-OS work through fixed-purpose runbooks, never a shell prompt The reality check SRE Agent hard-blocks delete / remove and all az keyvault commands, respects Azure management locks, and only one incident platform can be active at a time. Several patterns in this post need a custom tool or MCP server to complete the loop Time to first value One resource group, one alert rule, one response plan. You can reproduce use case #1 in an afternoon 1. Why the "investigation" half is the real prize Every conversation about AI in operations goes straight to remediation. "Will it restart my app?" That's the least interesting question, and it's the one with the most downside risk. Think about what actually consumes the minutes during a Sev1. The alert fires. Someone acknowledges. Then: Open Azure Monitor, confirm the metric is real and not a probe artifact Open Application Insights, find the dominant exception Open the deployment pipeline, find what shipped and when Open Activity Log, check whether someone changed configuration Open Resource Health, rule out a platform incident Open the other environment, confirm the previous version is healthy Assemble all of that into a sentence a human can act on That's twenty minutes of context assembly performed by a tired human, differently every time, with quality that depends entirely on who happens to be on call. It is the single most automatable part of incident response and the part nobody automates, because scripts can't reason about which of six hypotheses fits the evidence. This is exactly what root cause analysis in SRE Agent is designed for. The agent doesn't grep logs — it forms hypotheses and invalidates them: HYPOTHESIS 1: Recent deployment broke something ├─ Checked: Last deployment was 3 days ago ├─ Evidence: Error rate stable until 30 minutes ago └─ Result: INVALIDATED HYPOTHESIS 2: Database overloaded ├─ Checked: Azure SQL metrics (CPU, DTU, connections) ├─ Evidence: DTU at 98%, query duration 4x normal ├─ Traced: SELECT * FROM orders WHERE... taking 8.2s └─ Result: VALIDATED ROOT CAUSE: Orders table missing index on customer_id column. Query plan shows full table scan on 2.1M rows. RECOMMENDED ACTION: Add index on orders.customer_id Similar fix applied in INC-2341 (3 weeks ago) That last line — recalling a similar incident from three weeks ago — is the compounding part. Every thread produces a session insight: symptoms observed, steps that worked, root cause, and pitfalls to avoid. Thirty minutes after a thread goes quiet, the agent indexes those learnings. Next time the same resource misbehaves, that history surfaces first. So the framing for the rest of this post: remediation is the punchline, but investigation is the product. Every one of the ten use cases below has a large read-only phase you can turn on tomorrow with zero write permissions, and a small write phase you should gate behind approval for a long time. 2. What Azure SRE Agent actually does Before the use cases, here is the honest capability map, drawn from the product documentation rather than from a keynote. The three primary use cases Use case What it means Automate incidents Alert fires → agent queries monitoring tools, correlates signals across systems, identifies probable root cause, proposes mitigations Automate scheduled workflows Proactive health checks, compliance sweeps, and routine tasks on a schedule, with results routed to your incident platform or notification channel Investigate and advise Natural-language questions — "what changed in the last hour?" — answered with grounded citations The five extension primitives Everything you customize sits in one of five buckets: Primitive What it is When you reach for it Skills Procedural guidance ( SKILL.md ) plus optional attached tools; auto-loaded when relevant Team troubleshooting runbooks that should also execute Subagents / custom agents Purpose-built specialists invoked via /agent or routed to by a response plan A DatabaseExpert that owns every SQL incident Python tools Custom logic, transformations, API calls Anything that needs code, e.g. writing to the ServiceNow Table API MCP servers 40+ managed connectors (Datadog, New Relic, Splunk, Elastic, Dynatrace…) plus any custom MCP tool Bringing your non-Azure telemetry and your ITSM write surface into the loop Agent hooks Event-triggered automations at Stop and PostToolUse Policy enforcement, audit emission, blocking risky commands Six generic subagents ship built in — Explore, Plan, CodeReview, Bash, Verification, GeneralPurpose — and the agent can parallelize investigation, planning, review, shell, and verification work across them. A permission gate sits in front of all five primitives and evaluates every proposed tool call before it runs. Integrations you can assume Category Supported Monitoring Azure Monitor (metrics, logs, alerts, workbooks), Application Insights, Log Analytics, Grafana Incident platforms Azure Monitor Alerts, PagerDuty, ServiceNow — only one active at a time Source control / CI GitHub (repos, issues), Azure DevOps (repos, work items) Data Azure Data Explorer (Kusto), MCP servers Comms Slack, Microsoft Teams ⚠️ Design constraint worth internalizing early. Only one incident platform can be active at a time, and switching disconnects the current one. If your org runs PagerDuty for paging and ServiceNow for records of truth, you must pick which one the agent is bound to and reach the other through a connector or custom tool. Every use case below assumes ServiceNow is the bound platform. Skills vs. custom agents vs. knowledge files The three are constantly confused. This table settles it: Skills Custom agents Knowledge files Access Automatic when relevant Explicit ( /agent ) or routed by response plan Automatic search Tools Can attach Has its own None Context Uses thread context Shares thread context (no clean slate) Reference only Best for Team procedures with execution Domain specialists Runbooks, architecture docs Practical limits: a maximum of five concurrent active skills (oldest auto-unloads), knowledge base uploads up to 16 MB per file, and custom agent knowledge bases up to 1,000 files. 3. Anatomy of an agent-run incident Every use case in section 6 is an instance of this one shape. Learn it once. [[INCIDENT_FLOW_IMAGE]] Five properties make this shape safe, and they're worth stating as rules: The read phase has no blast radius. Turn it on everywhere, immediately, with Reader . One action per incident. Not "roll back and scale and restart." One bounded, reversible step, then re-measure. The proposed action is always the smallest reversible one. Swap a slot, don't redeploy. Bump one service tier, don't resize the cluster. Validation is a first-class phase, not a vibe. Define the metric, the threshold, and the duration before you approve. Technical recovery ≠ business recovery. Use case #10 makes this painfully clear. 4. Build the demo lab Everything below runs in a single throwaway resource group. Nothing here should touch a subscription you care about. 🧪 Lab hygiene. Create it, demo it, delete it. az group delete -n rg-sre-agent-demo --yes --no-wait when you're done. Several of these use cases deliberately break things. 4.1 Resource group and agent LOC=eastus2 RG=rg-sre-agent-demo az group create -n $RG -l $LOC Create the SRE Agent from the Azure portal and point it at $RG . Three things are created for you automatically: an Application Insights instance (this is where your audit trail lands), a Log Analytics workspace, a user-assigned managed identity (UAMI) — the identity every action runs as. 4.2 Choose a permission level deliberately At creation you pick one of two levels: Level Grants Use it when Reader Core monitoring roles + resource-type reader roles. Prompts for temporary elevation via on-behalf-of when it needs to act Start here. Production. Weeks 1–4 of any pilot Privileged Core monitoring roles + resource-type contributor roles Non-production, or after a proven pilot Regardless of level, these are always assigned: Role Scope Why Reader Resource group See resources and properties Log Analytics Reader Resource group Query logs and workspaces Monitoring Reader Resource group Read metrics Monitoring Contributor Subscription Acknowledge and close Azure Monitor alerts Grant additional access explicitly and narrowly: SUB=$(az account show --query id -o tsv) AGENT_MI=<agent-managed-identity-principal-id> # Read everywhere you want visibility az role assignment create \ --assignee $AGENT_MI \ --role "Reader" \ --scope "/subscriptions/$SUB" # Write ONLY where you intend the agent to act az role assignment create \ --assignee $AGENT_MI \ --role "Website Contributor" \ --scope "/subscriptions/$SUB/resourceGroups/$RG/providers/Microsoft.Web/sites/app-checkout-demo"</agent-managed-identity-principal-id> 📌 A sharp edge. You can't remove individual permissions from an agent — only entire resource groups. Removing a resource group from the agent's scope revokes all access to it. Plan your resource group boundaries as your blast-radius boundaries, because that's exactly what they are. 4.3 Connect ServiceNow Use a dedicated, least-privileged integration user — not a shared admin account. Method When What you need Basic auth Quick setup, PDI, testing Username + password, itil or admin role OAuth 2.0 Production ServiceNow OAuth app (client ID + secret); register redirect https://logic-apis-{region}.consent.azure-apim.net/redirect Scope the connection so you don't drown: Assignment group — essential on a shared enterprise instance Priority — Critical through Planning Category Scanner defaults worth knowing: Setting Value Scan interval 1 minute Incidents per page 20 Max incidents per cycle 220 (11 pages) Initial lookback 30 days Setup performs a real connectivity check by fetching an actual incident, so credential and endpoint mistakes surface immediately instead of six hours later when nothing syncs. 🚨 Delete the quickstart plan. Connecting an incident platform auto-creates a quickstart_handler response plan that runs in fully autonomous mode across all impacted services. If you then build your own plans, incidents get routed twice or to the wrong agent. Go to Builder → Incident response plans → Table view and delete it before you do anything else. 4.4 Connect deployment correlation Half the use cases below hinge on "what shipped three minutes before the spike." That correlation requires a source control connector: GitHub — repositories and issues Azure DevOps — repos and work items Connect one. Without it, the agent can still read Azure Activity Log and deployment history, but it can't reach commits, PRs, or work items. 4.5 Create the demo resources # 1 · App Service with a staging slot (use cases 1) az appservice plan create -g $RG -n plan-demo --sku P1V3 --is-linux az webapp create -g $RG -p plan-demo -n app-checkout-demo --runtime "DOTNETCORE:8.0" az webapp deployment slot create -g $RG -n app-checkout-demo --slot previous # 2 · AKS (use case 2) az aks create -g $RG -n aks-demo --node-count 3 --generate-ssh-keys --enable-addons monitoring # 3 · Azure SQL (use case 3) az sql server create -g $RG -n sqlsrv-sre-demo -u sqladmin -p "<use-a-generated-password>" az sql db create -g $RG -s sqlsrv-sre-demo -n db-customer --service-objective S1 # 4 · Cosmos DB (use case 4) az cosmosdb create -g $RG -n cosmos-sre-demo az cosmosdb sql database create -g $RG -a cosmos-sre-demo -n catalog az cosmosdb sql container create -g $RG -a cosmos-sre-demo -d catalog \ -n products --partition-key-path /category --throughput 400 # 5–7 · VMs (use cases 5, 6, 7) az vm create -g $RG -n vm-payments-linux --image Ubuntu2204 --size Standard_B2s --generate-ssh-keys az vm create -g $RG -n vm-claims-win --image Win2022Datacenter --size Standard_B2ms \ --admin-username azureadmin --admin-password "<use-a-generated-password>" # 8 · VM Scale Set (use case 8) az vmss create -g $RG -n vmss-api-demo --image Ubuntu2204 --instance-count 12 \ --upgrade-policy-mode automatic --generate-ssh-keys # 10 · Service Bus (use case 10) az servicebus namespace create -g $RG -n sb-sre-demo --sku Standard az servicebus queue create -g $RG --namespace-name sb-sre-demo -n order-events \ --enable-dead-lettering-on-message-expiration true</use-a-generated-password></use-a-generated-password> Install the Azure Monitor Agent on the VMs and associate a data collection rule — without guest telemetry, use cases 5, 6, and 7 have nothing to detect. 4.6 One custom agent per domain Don't build a single mega-agent. Build specialists and let response plans route to them: Custom agent Owns Attached tools DeploymentAnalyzer App Service, AKS, anything release-correlated RunAzCliReadCommands , GitHub connector, Kusto DatabaseExpert Azure SQL, Cosmos DB RunAzCliReadCommands , Kusto, read-only SQL diagnostics tool GuestOSResponder VM / VMSS / IIS RunAzCliReadCommands , fixed-purpose runbook tools only NetworkPathExpert Application Gateway, NSG, DNS, TLS RunAzCliReadCommands IntegrationExpert Service Bus, Event Hubs RunAzCliReadCommands , Kusto A custom agent definition is small: name: database_expert system_prompt: | You are a database specialist for this estate. Analyze query performance, diagnose connection and saturation issues, and recommend the smallest reversible mitigation. Never propose schema changes, index changes, plan forcing, or session termination — those are DBA-owned and require a change record. handoff_description: Handles Azure SQL and Cosmos DB troubleshooting tools: - execute_kusto_query - RunAzCliReadCommands allowed_skills: - azure-sql-saturation-runbook - cosmos-throughput-runbook Note what that system_prompt is really doing: it is narrowing the action space. Half of production safety with an agent is telling it, in plain English, which categories of fix are off the table. 4.7 Response plans Create one plan per domain, all starting in Review: Plan Filter Custom agent Mode appsvc-p1 Priority 1 + 2, service checkout DeploymentAnalyzer Review aks-p1 Priority 1, service orders-api DeploymentAnalyzer Review db-critical Priority 1 + 2, service customer-api DatabaseExpert Review vm-guest Priority 1 + 2, title contains disk , memory , CPU GuestOSResponder Review network-p1 Priority 1, title contains 502 , backend NetworkPathExpert Review integration-p1 Priority 1, service fulfillment IntegrationExpert Review Filters available: severity/priority (multiselect), impacted service, incident type, and title contains. Plans can be turned off without deleting them, which is exactly what you want during maintenance windows. 5. Set your guardrails before your first incident If you read only one section of this post, read this one. The guardrails are not paperwork; they are the reason this is deployable. 5.1 Run modes Mode Behavior Default Review Agent proposes; you approve or deny Agent-level default Autonomous Agent executes immediately and reports Per-plan and per-task default Two subtleties that bite people: Run modes are set per response plan and per scheduled task, not globally. The agent-level setting is only a fallback. And per-plan the default is Autonomous — so if you don't set it, you get autonomy. Review mode shows Approve/Deny only for Azure infrastructure operations (Azure CLI, ARM writes). Sending an email, posting to Teams, or querying an external source proceeds based on the agent's reasoning. To gate those, you need hooks or tool access policies. Only users holding the SRE Agent Administrator role can approve. Standard users cannot, and personal Microsoft accounts can't authorize on-behalf-of at all — it requires a work or school (Entra ID) account. 5.2 What the product blocks for you These are enforced at the command level, independent of your RBAC: Guardrail Behavior Delete operations The agent never runs delete or remove commands. It returns an error pointing you at the portal Key Vault All az keyvault commands are blocked, to prevent credential exposure Management locks Resources with ReadOnly locks can't be modified, regardless of permissions or run mode Subscription validation Subscription IDs are validated as well-formed GUIDs before execution 💡 Use the delete block architecturally. Put a ReadOnly management lock on anything that must never change during an incident — your Key Vaults, your production databases, your golden images. That lock is respected before every modification, which gives you a control that survives RBAC drift and misconfigured response plans. 5.3 Hooks: the guardrail you write yourself Two events are supported — Stop (agent about to return a final response) and PostToolUse (a tool finished). Hooks run as an LLM prompt or a sandboxed command script. Here is the single most useful hook for this entire post — a deterministic policy gate on shell execution: hooks: PostToolUse: - type: command matcher: "Bash|ExecuteShellCommand" timeout: 30 failMode: block script: | #!/usr/bin/env python3 import sys, json, re context = json.load(sys.stdin) command = context.get('tool_input', {}).get('command', '') dangerous = [ r'\brm\s+-rf\b', r'\bsudo\b', r'\bchmod\s+777\b', r'\bmkfs\b', r'\bdd\s+if=', r'\btruncate\b', r'\bDROP\s+TABLE\b', ] for pattern in dangerous: if re.search(pattern, command, re.IGNORECASE): print(json.dumps({"decision": "block", "reason": f"Blocked by policy: {pattern}"})) sys.exit(0) print(json.dumps({"decision": "allow"})) And a Stop hook that refuses to let the agent declare victory without evidence: hooks: Stop: - type: prompt model: ReasoningFast timeout: 30 prompt: | Review the agent's final response. $ARGUMENTS It is only acceptable if it contains ALL of: 1. The specific resource acted upon (full name or resource ID) 2. Metric values BEFORE and AFTER the action 3. The observation window over which recovery was confirmed 4. Who approved the action and at what UTC timestamp Respond with: {"ok": true} {"ok": false, "reason": "<what is="" missing="">"}</what> Hook mechanics you'll want on hand: Setting Default Range / notes timeout 30s 1–300s failMode allow allow or block maxRejections 3 1–25; prompt-type Stop hooks only matcher — Regex, anchored ^(pattern)$ , case-sensitive; * matches all Script size — 64 KB max Shebangs — #!/bin/bash , #!/usr/bin/env python3 ⚠️ The footgun: for Stop hooks, a rejection without a reason field is treated as approval. Always populate reason . Agent-level hooks and custom-agent-level hooks both run when both match; agent-level fires first. 5.4 The recommended production policy This is the policy I'd put in front of a change board: Allow read-only investigation automatically, everywhere. Allow automatic incident creation and work-note updates — after sanitization. Use Review mode for all production remediation. Require human approval for every production write. Use fixed-purpose runbooks instead of unrestricted VM shell access. Require separate approval for destructive or data-affecting actions. Initially require human confirmation before resolving an incident. 6. The ten use cases Each use case follows the same seven-part structure so you can skim to the one you're firefighting: What happened → How Azure Monitor detected it → How the agent found the cause → Agent action table → Recovery note → Try it yourself → Guardrails In the action tables, Type is one of: Type Meaning Read No blast radius. Safe to run automatically Decision Agent reasoning or an approval gate. No resource change Write (ITSM) Ticket create/update. Sanitized Write (Azure) Azure control-plane change. Approval required in production Write (Guest OS) Inside the VM, via a fixed-purpose runbook. Approval required Write (K8s/DevOps) Cluster or pipeline change. Approval required Validation Post-action measurement against defined thresholds Use case #1 — App Service: HTTP 500 after deployment The one you should build first. Clean trigger, bounded action, unambiguous validation. What happened Release 2026.03.12.4 was deployed to a production checkout App Service. The new code referenced an application setting that was never defined in the production slot. The application started fine — which is what makes this class of failure nasty — but every checkout operation that touched that setting returned HTTP 500. How Azure Monitor detected it Azure Monitor and Application Insights fired on a composite condition, not a single metric: HTTP 5xx rate above the operational threshold Failed availability tests Increased application exceptions Increased dependency failures Request latency above baseline P1 – Checkout App Service returning elevated HTTP 500 responses How the agent found the cause Identified the affected App Service and slot. Queried HTTP response and latency metrics. Queried Application Insights failed requests and exceptions. Identified the missing-setting exception as the dominant failure. Reviewed deployment history and Azure Activity Log. Correlated the error increase with release 2026.03.12.4 . Compared production against the previous deployment slot. Confirmed the previous slot remained healthy. Confirmed no matching Azure platform incident. Agent finding: HTTP 500 responses increased from 0.4% to 18% within three minutes of release 2026.03.12.4. Most failures reference a missing application setting. The previous deployment slot passes availability and dependency checks. Notice the shape of that sentence: a delta, a time correlation, and a known-good comparison. That's what makes it actionable rather than merely true. Agent action table # Action Type Detail 1 Investigate alert Read Reads HTTP 5xx rate, failed requests, latency, availability-test results 2 Analyze exceptions Read Identifies missing-configuration exception as dominant failure 3 Correlate deployment Read Finds release 2026.03.12.4 deployed three minutes before the spike 4 Compare slots Read Confirms previous slot healthy while production is failing 5 Rule out platform issue Read Checks Azure Resource Health and dependency health 6 Assess blast radius Read Determines checkout affected, unrelated services healthy 7 Create SNOW INC Write (ITSM) P1 against the Checkout CI, assigned to Application Operations 8 Update INC evidence Write (ITSM) Adds exceptions, deployment correlation, resource links, affected operations 9 Classify response Decision Classifies as deployment-caused; selects rollback 10 Pause release Write (DevOps) Pauses the failed release pipeline 11 Request approval Decision Requests approval to swap the previous healthy slot into production 12 Swap slot Write (Azure) Performs the approved swap only on the named App Service 13 Validate recovery Validation Confirms 5xx, latency, exceptions, dependencies, availability recover 14 Update/resolve INC Write (ITSM) Records approval, rollback, timestamps, recovery evidence 15 Create SNOW PRB Write (ITSM) Problem record for configuration-validation improvements Recovery note Production was reverted to the previous healthy deployment slot at 14:26 UTC. HTTP 500 responses declined from 18% to below 1%, and availability tests passed for 15 consecutive minutes. Preliminary cause: missing production configuration in release 2026.03.12.4. Try it yourself Break it: # Put a healthy build in the 'previous' slot first, then break production # by deploying code that reads an app setting which only exists in staging. az webapp config appsettings delete -g $RG -n app-checkout-demo \ --setting-names Checkout__PaymentProviderKey az webapp restart -g $RG -n app-checkout-demo Alert it: az monitor metrics alert create -g $RG -n "alert-checkout-5xx" \ --scopes "/subscriptions/$SUB/resourceGroups/$RG/providers/Microsoft.Web/sites/app-checkout-demo" \ --condition "total Http5xx > 20" \ --window-size 5m --evaluation-frequency 1m --severity 1 \ --description "P1 – Checkout App Service returning elevated HTTP 500 responses" Ask it (before the alert, to see the read phase in isolation): app-checkout-demo is returning HTTP 500s. Do not change anything. Investigate and tell me: 1. the dominant exception and its share of total failures 2. what deployed in the 30 minutes before the error rate changed 3. whether the 'previous' slot is healthy right now 4. whether Azure Resource Health shows a platform issue Give me the evidence chain and the smallest reversible mitigation. Expect it to propose: Proposed action: Swap slot 'previous' into production on app-checkout-demo Risk: brief connection drain (~10s). Fully reversible by swapping back. Validation: Http5xx < 1% and availability test passing for 15 minutes. [Approve] [Deny] Verify it: requests | where timestamp > ago(1h) | summarize failed = countif(success == false), total = count() by bin(timestamp, 1m) | extend failureRate = 100.0 * failed / total | render timechart 🎬 Best thing to record for a demo: the moment the Approve button appears with the evidence already attached. That single screen is the whole value proposition. Guardrails Scope write permissions to the named App Service, not the resource group. Verify the previous slot is healthy before swapping — a swap into a broken slot doubles the outage. Require human approval for production slot swaps. Preserve the failed deployment for analysis; don't let the pipeline overwrite it. Never grant subscription-level Contributor or Owner. Use case #2 — AKS: pods in CrashLoopBackOff What happened Image orders-api:4.18.0 referenced a configuration key that didn't exist in the production namespace. Eight of ten pods entered CrashLoopBackOff . The two survivors couldn't carry production traffic, producing latency and ingress 5xx. How Azure Monitor detected it Managed Prometheus / Container Insights detected: Unavailable replicas Increasing container restart count CrashLoopBackOff pod state Failed readiness and liveness checks Elevated ingress HTTP 5xx Reduced successful-request rate P1 – Orders API unavailable replicas in production AKS How the agent found the cause Identified the cluster, namespace, and deployment. Reviewed deployment availability and pod states. Examined logs from failing containers. Reviewed Kubernetes warning events. Compared current and previous ReplicaSets. Correlated the failure with revision 42. Identified the missing configuration key. Confirmed revision 41 was previously healthy. Checked node CPU, memory, storage, networking, status. Determined AKS infrastructure was healthy. Agent finding: Eight of ten Orders API pods entered CrashLoopBackOff during rollout of revision 42. Container logs show a missing configuration key. Revision 41, using image 4.17.6, was healthy. Step 10 is the one people skip. "Rule out the infrastructure" is what stops you from spending an hour on a node pool that was never the problem. Agent action table # Action Type Detail 1 Investigate workload Read Deployment status, unavailable replicas, readiness, restart counts 2 Analyze pod logs Read Startup failure from a missing configuration key 3 Analyze events Read Image pulls, mounts, scheduling, probes, container events 4 Correlate rollout Read Revision 42 deployed immediately before failure 5 Compare ReplicaSets Read Revision 41 was the previous healthy workload 6 Rule out infrastructure Read Nodes, memory, CPU, networking, storage healthy 7 Assess blast radius Read Eight of ten replicas unavailable 8 Create SNOW INC Write (ITSM) P1 against the Orders API CI 9 Update INC evidence Write (ITSM) Namespace, image, revision, errors, rollout correlation 10 Classify response Decision Application/configuration-caused 11 Request approval Decision Approval to pause and roll back revision 42 12 Pause failed rollout Write (K8s/DevOps) Prevents the release progressing or being reapplied 13 Roll back deployment Write (K8s) Rolls back only the named deployment to revision 41 14 Validate replicas Validation All expected replicas ready and stable 15 Validate application Validation Ingress errors decline; synthetic orders succeed 16 Update/resolve INC Write (ITSM) Revision, approval, rollback, recovery evidence 17 Create defect/task Write (ITSM) Add configuration validation to CI/CD Recovery note Orders API was rolled back from revision 42 to revision 41. All ten replicas are ready, restart counts have stabilized, ingress HTTP 5xx responses are below threshold, and synthetic order transactions are succeeding. Try it yourself Break it: az aks get-credentials -g $RG -n aks-demo kubectl create namespace orders kubectl create deployment orders-api -n orders --image=nginx:1.25 --replicas=10 kubectl rollout status deployment/orders-api -n orders # revision 41 equivalent, healthy # Now break it: point at an image whose entrypoint requires a missing env var kubectl set image deployment/orders-api -n orders orders-api=busybox:1.36 kubectl patch deployment orders-api -n orders --type=json -p='[ {"op":"add","path":"/spec/template/spec/containers/0/command", "value":["sh","-c","test -n \"$ORDERS_CONFIG_KEY\" || (echo \"FATAL: missing ORDERS_CONFIG_KEY\" >&2; exit 1); sleep 3600"]} ]' Watch it break: kubectl get pods -n orders -w kubectl rollout history deployment/orders-api -n orders Ask it: Pods in namespace 'orders' on aks-demo are crash looping. Read-only. Tell me: which revision introduced it, the exact container error, whether the node pool is healthy, and how many replicas are actually serving. Then tell me the last known-good revision and why you believe it was healthy. Expect it to propose: kubectl rollout undo deployment/orders-api -n orders --to-revision= scoped to that single deployment. Verify it: kubectl get deployment orders-api -n orders \ -o jsonpath='{.status.readyReplicas}/{.status.replicas}{"\n"}' kubectl get pods -n orders --no-headers | awk '{print $4}' | sort | uniq -c Guardrails Restrict access to the required cluster, namespace, and deployment. Do not grant cluster-admin . Require approval for rollback. Coordinate with GitOps reconciliation. If Flux or Argo owns that deployment, a kubectl rollout undo gets reverted within minutes and you've created a flapping outage. Either pause reconciliation first or roll back through Git. Do not permit namespace, persistent-volume, or cluster deletion. Use case #3 — Azure SQL Database: CPU saturation and timeouts The use case where the agent's job is to buy time, not to fix the problem. What happened A query execution plan changed after an application release. The new plan consumed substantially more CPU and workers. The database saturated, producing SQL dependency timeouts and failed customer requests. How Azure Monitor detected it Sustained CPU saturation Worker or session pressure Increased connection failures SQL dependency timeouts Query-duration deviation from baseline Degraded customer API success rate P1 – Customer API database saturation causing request timeouts How the agent found the cause Reviewed database CPU, workers, sessions, connections. Reviewed application SQL dependency failures. Checked for blocking and deadlocks. Ran approved read-only Query Store diagnostics. Identified the primary CPU-consuming query. Detected a recent execution-plan change. Correlated with an application release. Ruled out Azure service health and storage issues. Determined a temporary scale operation could restore service. Left permanent query remediation to the DBA team. Agent finding: Database CPU has remained saturated for 17 minutes. One query accounts for most recent CPU consumption and changed execution plan shortly before the incident. Application SQL dependency timeout rate is 23%. Step 10 is the design decision that makes this safe. The agent correctly diagnoses a plan regression and then deliberately does not fix it, because forcing a plan or dropping an index is a permanent, DBA-owned, change-controlled action. It buys capacity instead. Agent action table # Action Type Detail 1 Investigate database Read CPU, workers, sessions, connections, storage, availability 2 Analyze app impact Read SQL dependency latency, failures, affected API operations 3 Check blocking Read Approved diagnostics for blocking, deadlocks, connection growth 4 Analyze Query Store Read Highest-impact query and recent plan change 5 Correlate changes Read Recent application and database deployments 6 Rule out platform issue Read Service health, storage, database availability 7 Assess blast radius Read Affected applications; other databases healthy 8 Create SNOW INC Write (ITSM) P1 against the production database CI 9 Update INC evidence Write (ITSM) Utilization, query ID, timeouts, change correlation 10 Classify response Decision Scaling is mitigation; query changes remain DBA-owned 11 Calculate bounded scale Read Selects the smallest pre-approved capacity increase 12 Request approval Decision Database owner or incident commander 13 Scale database Write (Azure) Increases the affected database by one approved service step 14 Validate recovery Validation CPU, workers, timeouts, application success recover 15 Update INC Write (ITSM) Previous/new capacity, approval, timing, results 16 Create SNOW PRB Write (ITSM) Permanent query-remediation work 17 Create scale-down task Write (ITSM) Task/change to restore normal capacity after stability Recovery note Azure SQL capacity was temporarily increased by one approved service step. CPU declined from sustained saturation to 54%, and SQL dependency timeouts returned to baseline. Query Store indicates a probable execution-plan regression requiring permanent DBA remediation. Try it yourself Break it — generate a saturating workload against db-customer (an unindexed LIKE '%...%' scan in a tight loop from a container in the same region works fine on an S1). Alert it: az monitor metrics alert create -g $RG -n "alert-sql-cpu" \ --scopes "/subscriptions/$SUB/resourceGroups/$RG/providers/Microsoft.Sql/servers/sqlsrv-sre-demo/databases/db-customer" \ --condition "avg cpu_percent > 90" \ --window-size 5m --evaluation-frequency 1m --severity 1 \ --description "P1 – Customer API database saturation causing request timeouts" Ask it: db-customer is saturated. Read-only investigation. Identify the top CPU-consuming query, whether its plan changed recently, and what application release correlates. Do NOT propose index, schema, plan-forcing, or session-kill actions. Propose only the smallest temporary capacity step that restores service, and tell me what it costs per day and when we should scale back down. Read-only Query Store diagnostics the agent should run: SELECT TOP 10 qsq.query_id, qsp.plan_id, qsp.last_execution_time, SUM(qsrs.count_executions) AS executions, SUM(qsrs.avg_cpu_time * qsrs.count_executions) AS total_cpu_us FROM sys.query_store_query AS qsq JOIN sys.query_store_plan AS qsp ON qsp.query_id = qsq.query_id JOIN sys.query_store_runtime_stats AS qsrs ON qsrs.plan_id = qsp.plan_id JOIN sys.query_store_runtime_stats_interval AS qsrsi ON qsrsi.runtime_stats_interval_id = qsrs.runtime_stats_interval_id WHERE qsrsi.start_time > DATEADD(hour, -2, GETUTCDATE()) GROUP BY qsq.query_id, qsp.plan_id, qsp.last_execution_time ORDER BY total_cpu_us DESC; Expect it to propose: az sql db update -g $RG -s sqlsrv-sre-demo -n db-customer --service-objective S2 — exactly one step, on exactly that database. Guardrails Restrict scaling to the named database. Define minimum and maximum capacity. Require approval for scale-up and scale-down. Time-limit temporary capacity — an un-reversed emergency scale-up is how a P1 becomes a budget incident. Do not autonomously force plans, terminate sessions, modify indexes, or change schema. Track the temporary cost impact with an Azure Cost Management alert. Use case #4 — Azure Cosmos DB: HTTP 429 throttling The one where the correct root cause is "we're succeeding." What happened A marketing campaign increased Product Catalog traffic by roughly 40%. The Cosmos DB container hit its provisioned throughput ceiling. HTTP 429s increased, and client retries amplified application latency. How Azure Monitor detected it High normalized RU consumption Increased HTTP 429 responses Elevated server-side latency Application dependency failures Sustained operation near the throughput limit P2 – Cosmos DB throttling affecting Product Catalog requests How the agent found the cause Reviewed normalized RU consumption and throttled requests. Identified the affected database and container. Reviewed regional and partition behavior. Checked for hot-partition evidence. Reviewed application retry telemetry. Compared current traffic with the historical baseline. Correlated demand with the marketing campaign. Checked recent application deployments. Checked Azure service health. Determined the primary cause was legitimate demand. Agent finding: The Product Catalog container is at its configured throughput ceiling, and 21% of requests are being throttled. Traffic increased by approximately 40% following a campaign launch. No deployment or regional platform issue correlates with the event. Step 4 is the fork in the road. If consumption is uneven across partitions, more RU/s is money set on fire — the correct answer is an architecture change, not a scale-up. The agent has to check before it recommends. Agent action table # Action Type Detail 1 Investigate throttling Read RU consumption, 429 rate, latency, requests, availability 2 Identify scope Read Account, database, container, operations, regions 3 Analyze demand Read Compares traffic and RU consumption with historical patterns 4 Check partitions Read Looks for uneven partition consumption where telemetry permits 5 Analyze retries Read Determines whether client retries are amplifying the incident 6 Correlate events Read Links demand to campaign traffic; excludes release/platform issues 7 Assess blast radius Read Affected catalog operations; unaffected containers 8 Create SNOW INC Write (ITSM) P2 against the Product Catalog CI 9 Update INC evidence Write (ITSM) RU, throttling, latency, traffic, partition evidence 10 Classify response Decision Demand-driven unless hot-partition evidence exists 11 Calculate throughput Read Smallest increase within the cost ceiling 12 Request approval Decision Approval for a temporary throughput increase 13 Increase throughput Write (Azure) Raises throughput only to the approved maximum 14 Validate recovery Validation 429 rate and latency recover 15 Update/resolve INC Write (ITSM) Throughput, approval, cost implication, recovery 16 Create capacity task Write (ITSM) Work to return throughput to normal 17 Create SNOW PRB Write (ITSM) Partition or retry improvements, if required Recovery note Provisioned throughput was increased within the approved production limit. HTTP 429 responses declined from 21% to below 1%, and Product Catalog latency returned to baseline. The increase is temporary and will be reviewed after campaign traffic subsides. Try it yourself Break it: the container was created at 400 RU/s. Drive a few hundred reads per second at it and you'll be throttled within seconds. Alert it: az monitor metrics alert create -g $RG -n "alert-cosmos-429" \ --scopes "/subscriptions/$SUB/resourceGroups/$RG/providers/Microsoft.DocumentDB/databaseAccounts/cosmos-sre-demo" \ --condition "total TotalRequests where StatusCode == 429 > 100" \ --window-size 5m --evaluation-frequency 1m --severity 2 \ --description "P2 – Cosmos DB throttling affecting Product Catalog requests" Ask it: cosmos-sre-demo container 'products' is throttling. Read-only. Before recommending anything, tell me whether RU consumption is EVEN across physical partitions or concentrated. If it is concentrated, do not recommend a throughput increase — recommend an architecture problem record instead. If it is even, tell me the smallest RU/s that clears throttling and the daily cost delta. That prompt is the whole use case. Getting the agent to refuse the easy answer under a stated condition is the skill. Verify it: AzureDiagnostics | where ResourceProvider == "MICROSOFT.DOCUMENTDB" | where Category == "DataPlaneRequests" | summarize throttled = countif(statusCode_s == "429"), total = count() by bin(TimeGenerated, 1m) | extend throttleRate = 100.0 * throttled / total | render timechart Guardrails Define a maximum throughput ceiling the agent may not exceed. Restrict changes to the named container. Require approval for increases and reductions. Do not permit deletion, consistency-level changes, or region changes. Create cost alerts for prolonged increased throughput. Treat persistent hot partitions as an architecture issue, never as a scaling issue. Use case #5 — Azure VM: OS/root disk full The most operationally dangerous use case in this post, and the one with the most interesting guardrail design. What happened A legacy payment application generated excessive trace logs. Log rotation stopped working and the root filesystem filled. The VM stayed available at the Azure platform layer — heartbeat green, Resource Health fine — but the application stopped, because it could no longer write to disk. This is the classic "green dashboard, dead service" failure. Platform-layer monitoring alone will never catch it. How Azure Monitor detected it Azure Monitor Agent and guest telemetry detected: Critically low filesystem free space Rapid filesystem consumption Application process stopped Failed availability tests Elevated HTTP 5xx Healthy VM heartbeat but unhealthy application P1 – Legacy payment application unavailable due to full VM OS disk How the agent found the cause Confirmed the VM was online. Confirmed Azure Monitor Agent heartbeat. Identified the affected root/OS filesystem. Reviewed the free-space trend. Correlated application failure with disk exhaustion. Found 86 GB of growth in the application trace directory. Reviewed recent deployments and logging changes. Checked log-rotation status. Confirmed the files matched the approved cleanup policy. Excluded customer data, database files, audit logs, and system files. Agent finding: The VM is healthy at the Azure platform layer, but the OS volume has less than 1% free space. The approved application trace directory grew by 86 GB in six hours. The payment service stopped when it could no longer write to disk. Agent action table # Action Type Detail 1 Investigate VM Read VM, agent heartbeat, Azure platform health 2 Analyze filesystem Read Root volume and free-space trend 3 Correlate app failure Read Service stopped after disk exhaustion 4 Identify disk consumer Read Abnormal growth in an approved trace directory 5 Check changes Read Deployments, logging changes, rotation, scheduled tasks 6 Validate cleanup scope Read Candidate files meet approved path, type, and age rules 7 Protect data Decision Excludes databases, customer data, security logs, unknown files 8 Create SNOW INC Write (ITSM) P1 against the payment VM/application CI 9 Update INC evidence Write (ITSM) Disk usage, growth timeline, service impact, cleanup scope 10 Request approval Decision Approval for the restricted recovery runbook 11 Archive logs Write (Guest OS) Archives eligible files to protected Azure Storage 12 Remove eligible files Write (Guest OS) Removes only successfully archived, allowlisted files 13 Run log rotation Write (Guest OS) Executes the approved rotation operation 14 Restart service Write (Guest OS) Restarts only the named payment service if necessary 15 Validate recovery Validation Disk, application, archive, availability, growth stabilization 16 Update/resolve INC Write (ITSM) Bytes processed, exclusions, approval, results 17 Create SNOW PRB Write (ITSM) Permanent logging and rotation remediation Recovery note The approved recovery runbook archived and removed 82 GB of eligible application trace files. OS-volume free space is now 31%. The payment service was restarted and has passed health checks for 15 minutes. No database, customer, security, or system files were modified. That last sentence is not decoration. It is the sentence your auditor will read. Try it yourself Break it (lab VM only — this fills the root disk): az vm run-command invoke -g $RG -n vm-payments-linux \ --command-id RunShellScript --scripts " mkdir -p /var/log/payments/trace fallocate -l 24G /var/log/payments/trace/trace-$(date +%s).log df -h / " Ask it: vm-payments-linux root filesystem is nearly full and the payment service is down. Read-only first. Tell me: - exactly which directory grew, by how much, over what window - whether log rotation is configured and when it last ran - what changed in the last 24 hours that would explain it Then tell me which files are inside the approved cleanup allowlist (/var/log/payments/trace/*.log, older than 2h) and which are NOT, and confirm no database, audit, or customer data files are in scope. Do not delete anything. The critical design point. SRE Agent blocks delete and remove commands outright. You cannot have it rm those files through its Azure CLI surface, and you should be glad. The correct implementation is a fixed-purpose, version-controlled runbook that the agent invokes with tightly bounded parameters: # What the agent is allowed to call — one runbook, allowlisted parameters, nothing else az automation runbook start \ -g $RG --automation-account-name aa-sre-runbooks \ -n "Reclaim-TraceDiskSpace" \ --parameters vmName=vm-payments-linux \ allowedPath=/var/log/payments/trace \ pattern='*.log' \ minAgeHours=2 \ maxBytes=90000000000 \ archiveToContainer=payments-trace-archive \ requireArchiveBeforeDelete=true \ dryRun=false The runbook — not the agent — owns the destructive logic, and it enforces: path allowlist (refuse anything outside allowedPath ) filename pattern allowlist minimum file age maximum total bytes per execution successful archive verified before any deletion hard-refuse if any candidate file is unknown, or matches a protected pattern ( *.mdf , *.bak , /var/log/audit/* , *.key , *.pem ) a cooldown that prevents re-execution within N hours Verify it: az vm run-command invoke -g $RG -n vm-payments-linux \ --command-id RunShellScript \ --scripts "df -h /; systemctl is-active payments.service; ls -la /var/log/payments/trace | head" Guardrails Do not give SRE Agent unrestricted SSH or shell access. Ever. This is the single highest-leverage rule in this post. Use a version-controlled, fixed-purpose runbook. Allowlist paths, patterns, file ages, and maximum cleanup size. Stop if the responsible files are unknown. A disk filled by something you can't identify is a security event until proven otherwise. Require successful archival before deletion. Prevent repeated execution with a cooldown. Treat cleanup as temporary mitigation — the problem record is the fix. Use case #6 — Linux VM: anomalous CPU saturation Where "anomalous" is doing all the work. What happened Release 7.3.1 introduced an immediate retry loop when an inventory dependency failed. The application retried continuously without backoff, consuming nearly all VM CPU and causing request timeouts. How Azure Monitor detected it The alert deliberately combined conditions rather than firing on a threshold: CPU significantly outside the historical baseline Persistent saturation over multiple evaluations Increased request latency Increased dependency failures No approved maintenance or batch workload active P1 – Production order-processing Linux VM experiencing anomalous CPU saturation A static "CPU > 90%" rule on a batch-processing VM is a pager that everyone learns to ignore. The composite condition is what makes the alert worth waking someone — or an agent — for. How the agent found the cause Confirmed the VM was online. Compared current CPU with historical behavior. Identified the order-processing service as the main CPU consumer. Reviewed application request latency. Reviewed downstream dependency failures. Analyzed application retry logs. Correlated CPU growth with release 7.3.1. Excluded expected batch jobs. Excluded Azure maintenance or platform issues. Confirmed other application instances had sufficient capacity. Agent finding: CPU increased from a normal range of 35–50% to 98% four minutes after deployment 7.3.1. The order-processing service is repeatedly calling a failed dependency without backoff. No expected batch job or platform maintenance is active. Agent action table # Action Type Detail 1 Confirm anomaly Read Compares current CPU with the historical baseline 2 Analyze impact Read Latency, timeouts, availability, dependencies 3 Identify process Read Named order service is the primary CPU consumer 4 Correlate deployment Read Release 7.3.1, four minutes before saturation 5 Analyze logs Read Dependency retry loop without backoff 6 Rule out expected work Read Excludes batch, backup, maintenance, scheduled processing 7 Assess blast radius Read Traffic and available capacity on other instances 8 Create SNOW INC Write (ITSM) P1 against the Order Processing CI 9 Update INC evidence Write (ITSM) CPU, process, dependency, release, customer impact 10 Classify response Decision Rollback, not VM resize or arbitrary process termination 11 Request approval Decision Approval to drain, roll back, restart the service 12 Drain VM Write (Azure) Removes the VM from load-balancer rotation 13 Restore prior release Write (Guest OS/DevOps) Restores known-good version or configuration 14 Restart named service Write (Guest OS) Restarts only orders-service 15 Validate recovery Validation CPU, retries, latency, dependencies, health recover 16 Return to rotation Write (Azure) Restores traffic only after successful validation 17 Update/resolve INC Write (ITSM) Drain, rollback, restart, approval, results 18 Create defect Write (ITSM) Retry, backoff, and circuit-breaker remediation Steps 12 and 16 are the pattern to steal: drain before you touch, restore traffic only after validation passes. Most homegrown automation restarts a service while it's still taking traffic and turns a degradation into an outage. Recovery note Release 7.3.1 introduced a retry loop when the inventory dependency failed. After approval, the VM was drained, the previous release was restored, and orders-service was restarted. CPU declined from 98% to 43%, and application latency returned to baseline. Try it yourself Break it: az vm run-command invoke -g $RG -n vm-payments-linux \ --command-id RunShellScript --scripts " nohup bash -c 'while true; do curl -s -m 1 http://127.0.0.1:9/inventory >/dev/null 2>&1; done' & nohup bash -c 'while true; do :; done' & echo started " Ask it: CPU on vm-payments-linux is at 98%. Read-only. First: is this actually anomalous, or is it consistent with this VM's historical pattern for this hour and day of week? Show me the baseline. If anomalous: which process, which dependency is it calling, at what rate, and what deployed immediately before? Do NOT propose killing the top process or resizing the VM. Expect it to propose a drain → restore → restart sequence, with the drain step as a separate approval from the restart. Guardrails Do not automatically terminate the highest-CPU process. It is very often a legitimate workload, and occasionally it's a security incident you just destroyed the evidence for. Allow operations only for named services. Drain before service restart where possible. Escalate unknown or suspicious processes to security rather than remediating them. Limit restart attempts. Do not permanently resize the VM when the evidence points to faulty code. Resizing to survive a retry storm is buying hardware to host a bug. Use case #7 — Windows VM with IIS: memory leak What happened Release ClaimsPortal 5.9.0 introduced a memory leak in the Claims IIS application pool. Memory consumption climbed over several hours, paging began, request queues grew, and IIS returned HTTP 503. How Azure Monitor detected it Azure Monitor Agent collected available memory, committed bytes, paging activity, process working set, IIS request queues, HTTP 500/503 responses, and availability-test results. The alert required sustained abnormal growth, not a brief spike. P1 – Memory exhaustion affecting production IIS application How the agent found the cause Confirmed the VM and monitoring agent were healthy. Compared memory behavior with the historical baseline. Identified the relevant w3wp.exe process. Mapped it to the Claims application pool. Reviewed paging and request queues. Correlated HTTP 503 errors with low available memory. Reviewed Windows Event Logs. Correlated the growth with release 5.9.0. Excluded antivirus and scheduled-reporting activity. Confirmed another instance could carry traffic. Agent finding: Available memory declined from 42% to 4% over three hours. The Claims application pool grew from 1.8 GB to 11.6 GB without releasing memory after traffic normalized. Paging and HTTP 503 errors followed. The pattern began after release 5.9.0. "Without releasing memory after traffic normalized" is the sentence that distinguishes a leak from load. Cache growth under load is normal; failure to return afterwards is not. Agent action table # Action Type Detail 1 Confirm anomaly Read Memory, committed bytes, paging vs. baseline 2 Identify process Read Maps growing w3wp.exe to the Claims pool 3 Analyze IIS health Read Pools, queues, HTTP errors, availability 4 Correlate release Read Links sustained growth to release 5.9.0 5 Rule out other causes Read Excludes scheduled jobs, antivirus, maintenance 6 Assess capacity Read Confirms another instance can serve traffic 7 Create SNOW INC Write (ITSM) P1 against the Claims Portal CI 8 Update INC evidence Write (ITSM) Memory, paging, pool, release, HTTP impact 9 Classify response Decision Targeted recycling, not a full VM restart 10 Request approval Decision Approval to drain and recycle the named pool 11 Drain VM Write (Azure) Removes VM from load-balancer rotation 12 Capture diagnostics Write (Guest OS) Captures approved diagnostics to a protected location 13 Recycle app pool Write (Guest OS) Recycles only the Claims application pool 14 Validate recovery Validation Memory, paging, queues, HTTP errors, availability recover 15 Return to rotation Write (Azure) Restores traffic after health checks pass 16 Update/resolve INC Write (ITSM) Memory before/after, approval, stability 17 Create SNOW PRB Write (ITSM) Memory-leak remediation Step 12 before step 13 matters: recycling the pool destroys the evidence. Capture first. Recovery note Abnormal memory growth was isolated to the Claims application pool following release 5.9.0. The VM was drained, the approved application pool was recycled, and health checks passed before traffic was restored. Available memory increased from 4% to 61%, and HTTP 503 responses stopped. Try it yourself Break it (lab VM only): az vm run-command invoke -g $RG -n vm-claims-win ` --command-id RunPowerShellScript --scripts " Install-WindowsFeature Web-Server -IncludeManagementTools New-WebAppPool -Name 'ClaimsPool' # Simulate the leak \$leak = New-Object System.Collections.ArrayList 1..40 | ForEach-Object { [void]\$leak.Add((New-Object byte[] 100MB)) ; Start-Sleep -Milliseconds 200 } " Ask it: vm-claims-win available memory is at 4%. Read-only. Map the growing process to an IIS application pool. Show me the memory curve for the last 6 hours and tell me whether memory was released after traffic dropped. Correlate with deployment history. Then propose the most targeted possible mitigation — I do not want a VM restart. Expect it to propose: Restart-WebAppPool -Name "ClaimsPool" …and nothing else on that machine. Guardrails Recycle only the named application pool. Avoid full VM restart as the first response — it's a bigger hammer with a longer outage and it destroys the leak evidence. Do not copy memory dumps into ServiceNow. They contain credentials, tokens, and customer data. Store diagnostics in a secured location; put the link in the ticket. Prevent repeated automatic recycling — a pool that needs recycling every 40 minutes is an incident, not a routine. Escalate if the leak returns during the observation period. Use case #8 — Virtual Machine Scale Set: unhealthy instance The most autonomy-ready use case in the list, and the reason is stateless workloads. What happened A configuration extension failed while VMSS instance 17 was being provisioned. The VM was running but its application service never started. The instance failed application health probes and caused intermittent errors. How Azure Monitor detected it Reduced healthy backend count Application Health extension failure Backend health-probe failure VM extension provisioning failure Instance-specific errors Elevated intermittent HTTP 5xx P2 – Unhealthy VM Scale Set instance causing intermittent API failures How the agent found the cause Reviewed the VMSS healthy-instance count. Identified instance 17 as the only unhealthy instance. Reviewed backend health. Compared instance 17 with healthy instances. Checked the image and VMSS model. Reviewed VM extension state. Found the configuration extension failure. Reviewed boot and application diagnostics. Confirmed the workload was stateless. Confirmed eleven instances could carry production traffic. Agent finding: Instance 17 is the only unhealthy member of the 12-instance scale set. Its application health probe has failed since 09:18 UTC. The configuration extension failed during provisioning, and the application service never started. Eleven healthy instances can maintain service. Agent action table # Action Type Detail 1 Investigate VMSS Read Instance health, provisioning state, healthy count 2 Identify instance Read Instance 17 is the only unhealthy member 3 Analyze backend health Read Confirms the instance fails application probes 4 Compare instances Read Image, model, extensions, configuration 5 Find extension failure Read Locates the failed configuration extension 6 Review diagnostics Read Boot, extension, and application diagnostics 7 Assess safe capacity Read Eleven instances can carry traffic 8 Confirm statelessness Decision Replacement won't destroy required local state 9 Create SNOW INC Write (ITSM) P2 against the VMSS/application CI 10 Update INC evidence Write (ITSM) Instance, extension error, health, capacity evidence 11 Classify response Decision Replacement or reimage per approved procedure 12 Request approval Decision Approval to isolate and replace instance 17 13 Isolate instance Write (Azure) Ensures the instance receives no production traffic 14 Preserve evidence Read/Write Stores approved diagnostic evidence securely 15 Replace instance Write (Azure) Reimages or replaces only instance 17 16 Validate provisioning Validation Image, model, and extensions deploy successfully 17 Validate service Validation Backend health, capacity, customer errors recover 18 Update/resolve INC Write (ITSM) Replacement, approval, diagnostics, recovery 19 Create SNOW PRB Write (ITSM) Extension and image-validation improvement work Steps 8 and 7 are the gate. Replacing a stateless instance with eleven healthy peers is genuinely low risk. Replacing a stateful instance, or replacing one when you're already at minimum capacity, is an outage. Both must be confirmed, not assumed. Recovery note VMSS instance 17 was isolated after its configuration extension failed and the application service did not start. Diagnostic evidence was captured, and the instance was replaced after approval. The replacement passed extension, application, and backend health checks. The scale set has returned to 12 healthy instances. Try it yourself Break it: INSTANCE_ID=$(az vmss list-instances -g $RG -n vmss-api-demo \ --query "[5].instanceId" -o tsv) az vmss extension set -g $RG --vmss-name vmss-api-demo \ --name CustomScript --publisher Microsoft.Azure.Extensions \ --settings '{"commandToExecute":"exit 1"}' Ask it: vmss-api-demo has an unhealthy instance. Read-only. Identify which instance, since when, and the specific extension error. Confirm for me: (a) the workload is stateless, (b) how many healthy instances remain, and (c) whether remaining capacity can carry current traffic with 20% headroom. Only if all three are satisfied, propose a reimage of that single instance. Capture diagnostics before proposing anything. Expect it to propose: az vmss reimage -g $RG -n vmss-api-demo --instance-id $INSTANCE_ID Verify it: az vmss get-instance-view -g $RG -n vmss-api-demo --instance-id $INSTANCE_ID \ --query "vmHealth.status.code" az vmss list-instances -g $RG -n vmss-api-demo -o table Guardrails Confirm the workload is stateless before any replacement. Confirm sufficient healthy capacity first. Preserve diagnostic evidence before replacement — the instance is your only copy of the failure. Restrict permissions to the named VMSS. Limit simultaneous replacements. One at a time. An agent that reimages six instances because six probes failed has just caused the outage it was investigating. Do not permit deletion of the entire scale set. Use case #9 — Application Gateway: HTTP 502 from unhealthy backends The cross-component change nobody coordinated. What happened Customer Portal release 9.2 changed the backend listener from port 443 to 8443. Application Gateway remained configured to connect on 443. All backend probes failed and customers received HTTP 502. How Azure Monitor detected it Increased Application Gateway HTTP 502 responses Increased failed requests Four unhealthy backends Reduced healthy-host count Failed synthetic availability tests P1 – Application Gateway returning HTTP 502 due to unhealthy backends How the agent found the cause Reviewed Application Gateway metrics. Retrieved backend health. Identified the affected pool. Reviewed probe path, protocol, host header, and port. Reviewed backend settings. Confirmed the application responded on 8443. Confirmed the gateway used 443. Correlated the mismatch with release 9.2. Reviewed NSG, route, DNS, certificate, and WAF changes. Excluded networking, certificate, and platform-health issues. Agent finding: HTTP 502 responses began at 16:07 UTC. All four Customer Portal backends are unhealthy. Release 9.2 changed the backend listener to port 8443, while Application Gateway continues to use port 443. No NSG, routing, or certificate issue correlates with the incident. Step 9 is what separates a real investigation from a lucky guess. A 502 has at least six plausible causes — NSG, UDR, DNS, expired cert, WAF rule, backend down. The agent has to exclude them, in writing, before you trust the conclusion. Agent action table # Action Type Detail 1 Investigate gateway Read HTTP 502, failed requests, latency, backend counts 2 Inspect backend health Read Identifies four unhealthy Customer Portal backends 3 Review configuration Read Settings, probes, protocol, port, TLS, routing 4 Test backend state Read App responds on 8443 while gateway uses 443 5 Correlate changes Read Links mismatch to Customer Portal release 9.2 6 Rule out networking Read NSGs, routes, DNS, TLS, WAF, platform health 7 Assess blast radius Read Confirms the Customer Portal pool is unavailable 8 Create SNOW INC Write (ITSM) P1 against the portal/gateway CI 9 Update INC evidence Write (ITSM) 502s, backend, port, deployment, known-good configuration 10 Classify response Decision Restore the known-good backend listener 11 Request approval Decision Approval to restore source-controlled configuration 12 Restore listener Write (Application) Restores the application listener to approved port 443 13 Validate backend health Validation All four backends become healthy 14 Validate application Validation HTTP 502 declines; synthetic transactions pass 15 Verify controls Read Confirms no WAF, TLS, routing, or NSG control was weakened 16 Update/resolve INC Write (ITSM) Cause, restoration, approval, recovery 17 Create SNOW CHG Write (ITSM) Coordinated change for the intended port migration 18 Create SNOW PRB Write (ITSM) Cross-component deployment-validation work Step 12 is a genuinely important choice. There were two ways to fix this: change the application back to 443, or change the gateway to 8443. The agent restores the application to the known-good, source-controlled state rather than mutating the gateway to match an unapproved change. One of those is a rollback; the other is ratifying an unreviewed change during an outage. Then step 17 files a proper change record for the migration the team clearly intended to do. Step 15 exists because the fastest way to make a 502 disappear is to disable TLS validation. The agent must prove it didn't take the fast way. Recovery note Customer Portal release 9.2 changed the backend listener from port 443 to 8443 without a coordinated gateway change. The application listener was restored to the previous configuration. All four backends are healthy, HTTP 502 responses returned to baseline, and synthetic login tests passed for 15 minutes. Try it yourself Ask it: appgw-portal is returning 502s and all backends are unhealthy. Read-only. Walk me through the exclusion, explicitly, for each of: NSG, UDR/route table, DNS resolution, backend TLS certificate, WAF rule blocking, backend process down, and probe configuration mismatch. State which you ruled out and the evidence for each. Then tell me the known-good configuration and where it is source-controlled. Verify it: az network application-gateway show-backend-health \ -g $RG -n appgw-portal \ --query "backendAddressPools[].backendHttpSettingsCollection[].servers[].{addr:address,health:health}" -o table Guardrails Do not disable TLS validation. Not to test, not temporarily, not "just to confirm." Do not weaken NSGs or WAF policies as a mitigation. Use source-controlled configuration as the definition of "known-good." Restore a known-good state during the incident; migrate through a change record afterwards. Require approval for gateway or backend changes. Validate full application transactions, not just health probes. A probe returning 200 on /health proves very little. Use case #10 — Azure Service Bus: queue and dead-letter backlog The one that teaches the most important lesson in the entire post. What happened Release fulfillment-worker:6.4.0 couldn't deserialize messages containing a new deliveryWindow field. Consumer throughput dropped by 92%. The active backlog grew rapidly, and incompatible messages entered the dead-letter queue. How Azure Monitor detected it Increasing active-message count Increasing oldest-message age Dead-letter growth Reduced completed-message rate Consumer application errors Delayed downstream business processing P1 – Production order-event backlog delaying fulfillment How the agent found the cause Reviewed active, incoming, outgoing, and dead-letter counts. Calculated message arrival and completion rates. Confirmed the backlog was growing. Reviewed consumer instance health. Reviewed consumer errors and restarts. Checked downstream dependency health. Checked Service Bus authentication and authorization. Correlated the throughput decline with release 6.4.0. Found deserialization errors for the new field. Determined scaling more broken consumers would amplify failures. Agent finding: The order-events queue grew from 3,000 to 185,000 active messages in 35 minutes. Consumer throughput dropped by 92% immediately after release fulfillment-worker:6.4.0. Application logs show deserialization failures involving the new deliveryWindow field. Step 10 is the whole reason to use a reasoning agent instead of an autoscale rule. Every metric here screams "scale out the consumers." An HPA would have done exactly that, and every new replica would have dead-lettered messages faster. Agent action table # Action Type Detail 1 Investigate queue Read Active, incoming, outgoing, scheduled, DLQ counts 2 Calculate flow rates Read Confirms arrival exceeds completion; backlog growing 3 Inspect consumers Read Health, instance count, scaling, errors, dependencies 4 Analyze failures Read Deserialization exceptions involving the new field 5 Correlate release Read Links the 92% throughput reduction to release 6.4.0 6 Rule out Service Bus Read Health, authorization, throttling, networking, service status 7 Assess blast radius Read Backlog age and fulfillment impact 8 Protect message data Decision Prevents payloads or personal data entering ServiceNow 9 Create SNOW INC Write (ITSM) P1 against the fulfillment integration CI 10 Update INC evidence Write (ITSM) Backlog, age, flow, exception, release, business impact 11 Classify response Decision Rollback, not scaling broken consumers 12 Request approval Decision Approval to restore consumer 6.3.7 13 Roll back consumer Write (Azure/K8s) Rolls back through the approved deployment platform 14 Restore capacity Write (Azure/K8s) Restores approved consumer instance count 15 Validate processing Validation Consumer and downstream health 16 Validate backlog Validation Completion exceeds arrival; DLQ growth stops 17 Update INC Write (ITSM) Rollback, rates, estimated drain time, approval 18 Maintain incident Decision Keeps the incident open until backlog age meets the objective 19 Create SNOW CHG Write (ITSM) Separate controlled change for DLQ replay 20 Create SNOW PRB Write (ITSM) Message-contract compatibility remediation Recovery note Consumer release 6.4.0 could not deserialize messages containing the new deliveryWindow field. The fulfillment worker was rolled back to 6.3.7. Consumer throughput recovered, new dead-letter growth stopped, and the active backlog is draining at approximately 7,500 messages per minute. Dead-letter replay requires a separately approved procedure. The lesson: technical recovery is not business recovery Step 18 is the most important row in this entire post. At the moment of rollback, every technical signal is green. Consumers are healthy. Throughput has recovered. The DLQ has stopped growing. An agent optimizing for metrics would resolve the incident right there and go back to sleep. But there are still 185,000 unshipped orders and a dead-letter queue full of messages that need a separately approved replay procedure. Customers are still affected. The incident stays open until backlog age meets the business objective, not until the graphs look nice. Encode this in the response plan explicitly: Do not resolve this incident when consumer health recovers. Resolution criteria: 1. Completion rate exceeds arrival rate for 15 consecutive minutes, AND 2. Oldest active message age is under 5 minutes, AND 3. Dead-letter count has not increased for 30 minutes. DLQ replay is out of scope for this incident. File a separate change record. Try it yourself Break it: # Flood the queue while no consumer is running for i in $(seq 1 5000); do az servicebus queue message send -g $RG --namespace-name sb-sre-demo \ -q order-events --body "{\"orderId\":$i,\"deliveryWindow\":\"2026-08-09T10:00Z\"}" 2>/dev/null done az servicebus queue show -g $RG --namespace-name sb-sre-demo -n order-events \ --query "countDetails" -o json Ask it: order-events on sb-sre-demo has a growing backlog. Read-only. Give me: arrival rate, completion rate, current active count, oldest message age, DLQ count and DLQ growth rate, and the projected drain time at current rates. Then tell me why scaling out consumers is or is not the correct action here. Do not include any message payloads or customer data in your answer. That last line is not optional. Message bodies routinely contain names, addresses, and payment references — and everything the agent writes goes into a ticket. Verify it: az servicebus queue show -g $RG --namespace-name sb-sre-demo -n order-events \ --query "{active:countDetails.activeMessageCount, dlq:countDetails.deadLetterMessageCount}" -o json Guardrails Never automatically purge queues. Do not automatically replay dead-letter messages. Replay without idempotency guarantees means duplicate charges and duplicate shipments. Do not place message payloads in ServiceNow. Confirm idempotency before replay. Scale consumers only when the processing path is healthy. Require separate approval for replay or queue configuration changes. Keep the incident open until business recovery is confirmed. 7. The ITSM integration model Recommended incident fields Field Example Short description Production Orders API pods failing after deployment Configuration item prod-aks-orders-api Assignment group Container Platform Operations Impact High Urgency High Environment Production Azure resource ID Full affected Azure resource ID Azure alert ID Azure Monitor alert correlation identifier SRE investigation Link to the SRE Agent investigation thread Current impact Eight of ten replicas unavailable Probable cause Missing configuration in latest release Confidence High Proposed action Roll back to revision 41 Approval Approver and UTC timestamp Validation Ten replicas ready and synthetic tests passing Resolution Service restored through rollback The Confidence field earns its place. An agent that says "probable cause: missing configuration (confidence: low)" is far more useful than one that always sounds certain, because it tells the human how much to verify before approving. Fields the agent can set directly (preview): assignment_group , category , subcategory , impact , urgency , priority , short_description , and any custom u_* field. It cannot change incident state through field updates — acknowledge and resolve are separate, dedicated tools. Recommended agent-generated timeline Every work note should carry: Element Example Timestamp 2026-03-18 14:26 UTC Observation HTTP 500 rate increased to 18% Evidence Link to the Application Insights query Change correlation Incident started three minutes after deployment Probable cause Missing production application setting Confidence High Proposed action Swap to previous healthy slot Risk Temporary deployment rollback Approval Incident commander and timestamp Execution Slot-swap operation and result Validation 5xx below 1% for 15 minutes Follow-up Problem record for configuration validation Note that Evidence is a link, not a paste. This is a deliberate data-protection pattern: the ticket carries a pointer to the query, and the query results stay in the system that already has the right access controls. Deduplication strategy Do not create one incident per alert, pod, queue, or VMSS instance. Correlate on: application/business service + Azure resource ID + environment (production) + alert-rule family + active incident time window Related alerts attach to the existing incident as evidence or child alerts. Azure Monitor already merges recurring alerts into a single thread when it's the bound platform; for ServiceNow, this correlation key is yours to implement. Get this wrong and your first AKS incident produces eight incidents, eight investigations, and eight rollback proposals for the same deployment. Record responsibilities Record Purpose Example Incident (INC) Restore service quickly and safely App Service HTTP 500 outage Problem (PRB) Identify and remove the underlying cause Missing deployment configuration validation Change (CHG) Govern permanent or higher-risk production changes Coordinated Application Gateway port migration Engineering defect/task Correct application or automation behavior Add retry backoff to the Linux application The agent must clearly distinguish temporary mitigation from permanent correction. Every single use case above ends with a follow-up record, and that's not bureaucratic theatre — an agent that mitigates flawlessly and never files a problem record is an agent that lets the same outage recur forever while making the metrics look great. 8. Reality check: where you have to build This is the section that will save you a month. The PDF this post is built from is explicit that these are target response patterns, not guaranteed zero-configuration behavior. Having now checked each pattern against the product documentation, here is exactly where the gaps are and what fills them. # The pattern assumes What's actually documented What you must build 1 Agent creates a ServiceNow INC ServiceNow is an inbound platform. Documented writes: post discussion entries, acknowledge, resolve, plus field updates (preview) Incidents should originate in ServiceNow (via its own Azure Monitor integration) and flow in. If you truly need agent-initiated creation, add a Python tool or MCP server against the ServiceNow Table API 2 Agent creates PRB / CHG / defect records Not a documented first-class ServiceNow action Same: a custom tool against /api/now/table/problem and /change_request . This is ~30 lines of Python and worth doing properly, with a least-privileged integration user 3 ServiceNow and PagerDuty both connected Only one incident platform active at a time; switching disconnects the other Bind the agent to your system of record (ServiceNow). Reach the pager through a connector, Teams/Slack, or a webhook 4 Agent runs guest-OS cleanup ( rm , rotate, restart) delete and remove commands are blocked outright. az keyvault blocked. Management locks respected Wrap all guest-OS work in fixed-purpose Azure Automation runbooks or a constrained az vm run-command script, invoked with allowlisted parameters. See use case #5 5 Agent "pauses the release pipeline" Requires the GitHub or Azure DevOps connector, plus permissions on that pipeline Connect source control; grant pipeline permissions explicitly; test the pause path before you need it 6 Approval gate on every action Review mode shows Approve/Deny only for Azure infrastructure operations. Emails, Teams posts, and external queries proceed on the agent's reasoning Use hooks or tool access policies to gate non-Azure actions 7 Agent resolves incidents Supported — but during a pilot you don't want it Require human confirmation before resolve. Encode it in the response plan and enforce it with a Stop hook 8 One agent handles everything Response plans route to custom agents; skills cap at five concurrent active Build domain specialists (§4.6). A single mega-agent thrashes its skill budget 9 Autonomous mode by default is fine Per-plan default is Autonomous, and connecting a platform auto-creates an autonomous quickstart_handler Delete the quickstart plan. Set every plan to Review explicitly 10 Agent has broad subscription rights You can't remove individual permissions — only whole resource groups Design resource groups as blast-radius boundaries before onboarding None of these are blockers. All of them are a week of work you'd rather discover now than during your pilot readout. 9. Approval and autonomy policy The policy I would actually ship: Action category Recommended initial policy Read metrics, logs, traces, resource health Automatic Correlate deployments and configuration changes Automatic Create a ServiceNow incident Automatic after deduplication Add sanitized work notes Automatic Prepare a remediation plan Automatic Create a draft ServiceNow change Automatic, but not approve it Modify an Azure production resource Human approval required Execute a VM guest runbook Human approval required Roll back an application Human approval required Delete or replace a stateless VMSS instance Human approval required Resolve a ServiceNow incident Human confirmation during the pilot Delete data, purge queues, replay DLQ messages Separate explicit approval Autonomous remediation Only after a proven, bounded pilot Two notes on making this real: Start in Review and stay there longer than feels necessary. The documented recommendation is to observe for two to four weeks and then promote specific triggers you consistently approve. Not the agent — the triggers. Promotion should be per-response-plan and evidence-based: "we approved this exact rollback proposal eleven times without modification" is a reason to go autonomous. "It seems good" is not. Autonomy should be earned per action type, not per environment. "Autonomous in staging" is a fine starting rule, but the durable version is "autonomous for VMSS single-instance reimage where the workload is stateless and healthy capacity exceeds 80%" — a narrow, well-characterized action with a mechanical precondition. 10. Cross-cutting security controls Use a dedicated managed identity for SRE Agent. Scope Azure roles to selected resources or resource groups. Avoid broad Contributor and Owner assignments. Use fixed-purpose Automation runbooks for guest operations. Do not provide unrestricted SSH, shell, or PowerShell execution. Use a dedicated least-privileged ServiceNow integration identity. Prefer OAuth or a managed connector over stored credentials. Store required secrets in Key Vault — never in prompts. (The agent blocks az keyvault commands entirely, which helps.) Sanitize logs before posting them into ServiceNow. Do not post tokens, personal data, SQL text, message payloads, or memory dumps. Record every approval and production action. Define remediation cooldowns and maximum retry counts. Require post-action application validation. Retain existing manual runbooks as fallback. Use Azure Cost Management alerts for temporary scaling actions. What the platform gives you for free Worth knowing so you don't rebuild it: Layer Isolation model Compute Dedicated sandbox (micro VM) per agent; tool execution separate from the reasoning loop Database Separate database per agent Blob storage Separate blob storage per agent Network Per-agent proxy instance validating every outbound request Credentials Identity sidecar issues short-lived, per-call tokens; credentials never enter the reasoning context Token lifetimes: managed identity ~1 hour (auto-refreshed), OAuth refreshed 20 minutes before expiry, per-tool-call action tokens are single-use, blob SAS 1 hour refreshed at 45 minutes. Each tool invocation launches a fresh process whose entire tree terminates on completion — there are no persistent process pools, so one tool call cannot see another's environment. The audit trail Every az command is logged to your Application Insights as an AgentAzCliExecution custom event. This is your evidence for change management: customEvents | where name == "AgentAzCliExecution" | where timestamp > ago(30d) | project timestamp, command = tostring(customDimensions.command), resource = tostring(customDimensions.resourceId), succeeded = tostring(customDimensions.success), thread = tostring(customDimensions.threadId) | order by timestamp desc Run that query in front of your auditor once and most of the "but can we prove what it did" conversation ends. 11. A 30/60/90 pilot that survives contact with your CAB Phase Enabled capability 1 Detect Azure Monitor alerts 2 Create or correlate ServiceNow incidents 3 Perform read-only investigation 4 Add sanitized findings to ServiceNow 5 Recommend remediation without execution 6 Execute bounded actions after approval in Review mode 7 Validate technical and business recovery 8 Prepare incident resolution and follow-up records 9 Consider autonomy only for proven low-risk actions Phases 1–5 have zero production write risk and deliver most of the MTTR reduction. Do not rush past them to get to the demo-friendly part. The best first five candidates App Service deployment-slot rollback — clean trigger, reversible action, unambiguous validation AKS deployment rollback — same shape, one GitOps caveat VMSS unhealthy-instance replacement — stateless, bounded, easy precondition check Restricted VM disk-recovery runbook — high toil, high value, forces you to build the runbook pattern properly Automatic ServiceNow incident creation and timeline updates — the compounding one; every incident from here on is better documented than any incident before it These five share the properties you want: clear triggers, tightly bounded actions, measurable validation criteria, and practical escalation paths. What to measure in week one Before you enable a single write action, capture your baseline: Median time from alert to first accurate human diagnosis Percentage of incidents where the first hypothesis was wrong Median time from diagnosis to mitigation Percentage of incidents with a complete timeline in the ticket Percentage of incidents that produced a follow-up problem record The read-only phase moves the first, second, and fourth of those immediately. If it doesn't, your telemetry is the problem, not the agent — and that's a genuinely useful thing to discover in week one rather than week twelve. 12. Measuring whether it's actually working Under Monitor → Incident metrics: Metric What it shows Incidents reviewed Total incidents the agent processes Mitigated by agent Resolved autonomously Assisted by agent Agent helped; a human completed it Mitigated by user Human resolved using agent-provided information Pending user action Waiting on a human The counter-intuitive read: "Assisted by agent" and "Mitigated by user" are the healthy numbers during a pilot. A high "Mitigated by agent" count in month one means someone left autonomy on. Watch Pending user action closely. A growing queue there means either your approval routing is broken or the agent is proposing things nobody is comfortable approving — both are important signals, and both are invisible without this dashboard. Also check Monitor → Session insights periodically. Each insight card links back to the thread that generated it, so you can trace any learned pattern to its origin. If the agent has learned something wrong, this is where you find it — and #forget is how you fix it. 13. Resources Core documentation Overview of Azure SRE Agent Security and trust model Agent permissions Run modes Incident response Incident management platforms Incident response plans ServiceNow incident indexing Root cause analysis Execute mitigations Extensibility Custom agents Skills Agent hooks Scheduled tasks Memory and knowledge Closing The framing that makes this work isn't "AI runs my production." It's this: Your agent is the most junior person on the rotation — and the most thorough. It will never skip the Resource Health check. It will never forget to compare against the previous slot. It will never write "restarted it, seems fine" in a work note at 4 AM. And it will never, ever be allowed to rm -rf anything. Every guardrail in this post exists to keep it in that role. Scope the identity to resource groups. Keep production writes in Review. Wrap guest-OS work in runbooks with allowlists. Never let it purge a queue or replay a dead-letter message on its own. Keep the incident open until customers are actually served, not until the graphs look nice. Do that, and the ten workflows above stop being a slide deck and start being your Tuesday. Start with use case #1. One App Service, one slot, one alert rule, one response plan in Review mode. Watch it assemble an evidence chain you'd have spent twenty minutes building by hand, and then decide how much further you want to go. The ten scenarios in this post are target response patterns. Each one requires appropriate telemetry, scoped Azure RBAC, response-plan instructions, approved remediation tooling, and ITSM integration. Confirm current Azure SRE Agent and ServiceNow connector capabilities against Microsoft documentation before implementing — the product is moving quickly, and several capabilities referenced here are in preview.
