From Alert to Resolved: Building a Self-Healing Azure Platform with SRE Agent
Hello Folks! It’s 3 a.m. Your phone lights up. A critical workload that spans multiple clouds is on fire, ownership is fuzzy, the alert routed to the wrong team first, and now it’s your problem. You sit up in bed, cold and groggy, and start the ritual. Open the runbook. Pull logs from one place. Pull metrics from another. Stare at three dashboards. None of them tell the whole story. So you build a theory. The theory is wrong. The clock keeps ticking. The customer impact keeps climbing. Every wrong turn costs you time, context, and confidence. That is the scene Lee Oommen opened with at MAIS14, and it is the reason Azure SRE Agent exists. In this session, Lee walks through the four classic SRE pain points and shows how an agentic operations platform compresses MTTR from hours to minutes. I am going to unpack what he showed, why it matters for IT pros, and how to get your hands on it. Why IT Pros Should Care If you carry a pager, write runbooks, or get pulled into post-mortems, this one is for you. The clock is the enemy, not the incident. Lee said it plainly: there is almost always an expert who can fix the problem. The real damage comes from the minutes spent finding that expert and reconstructing context. Dashboards lie in both directions. False positives create alert fatigue. False negatives let the customer call you before your monitors do. Neither outcome is acceptable. RCAs take weeks because the answer never lives in one layer. Infrastructure, network, deployments, dependencies, databases, app code. You need someone, or something, that can correlate across all of them in one pass. You did not become an SRE to be a dashboard watcher. Toil is the work that holds back the people who should be designing reliability into the next generation of services. What is the SRE Agent Let’s start with what it is not. It is not a dashboard. It is not a monitoring tool. It is not a chatbot. Azure SRE Agent is an end-to-end agentic operations platform. Think of it as a senior SRE who sits inside your team, works 24 by 7, never gets tired, never misses a signal, and is fluent in your stack. Reasons over telemetry, not just text. It pulls metrics, logs, traces, deployment history, and activity logs, then correlates across them. Takes governed actions. Every action runs inside the permission boundary you define. You decide whether the agent proposes a fix, asks for approval, or acts autonomously. Authors RCAs in minutes, not weeks. It traces the root cause in a single flow across the entire stack and produces the report immediately after remediation. Remembers. It captures organizational memory from every incident, every chat, and every scheduled task, then applies it to the next investigation. Lee called it the operations half of DevOps, and that framing stuck with me. We have automated build and deploy. The operations side has been stuck in toil. SRE Agent closes that loop. From Alert to Resolved (the workflow) Lee demonstrated the full loop live. Here is what it looks like end to end. Detect. The agent integrates with Azure Monitor, PagerDuty, or ServiceNow. When an alert fires, the agent acknowledges it within seconds. The human does not have to wake up cold. Investigate. The agent runs diagnostics in parallel across the connected resources. It queries Log Analytics, App Insights, Azure Monitor metrics, and any third-party observability tools you wired in through MCP connectors. Correlate. It uses distributed tracing, cross-workspace KQL queries, and time-based signal alignment to connect dots across services that do not even share trace IDs. It also checks past incidents in memory to see if this looks familiar. Diagnose. It produces a root cause analysis with the relevant evidence linked inline. No more reconstruction exercise across multiple teams. Propose or Act. Based on your run mode and the permissions granted, the agent either proposes a fix and waits for approval, or executes the remediation autonomously. Lee demonstrated both. He set up a bad slot swap on Azure App Service, generated HTTP 500 errors, watched the agent acknowledge the alert, investigate, identify the bad slot, ask for permission, and then perform the slot swap to restore the service. Close the loop. The agent files a GitHub or Azure DevOps issue with full context, opens a pull request with proposed code changes when appropriate, and writes a session insights summary you can review. Three modes to interact with it: Interactive. Chat with the agent like a copilot. Most customers start here to build trust. Reactive. Event-driven. The agent reacts to incidents from Azure Monitor, PagerDuty, or ServiceNow. One agent per incident platform. Proactive. Scheduled tasks that run every five minutes, every hour, daily, or weekly. Certificate health audits, well-architected framework assessments, cost optimization sweeps, compliance checks. Lee showed a scheduled task that flagged a certificate expiring in 50 days before it could ever fire an alert. Real-World Value This is where the conversation gets practical. A few things from Lee’s demo and the live Q&A that I want to call out. Multi-cloud reality. SRE Agent lives in Azure but is not limited to Azure. Custom runbooks, Python execution, MCP servers, and connectors let it orchestrate across AWS, GCP, and on-premises. Treat it as the central SRE brain. Your data stays yours. Each agent gets a dedicated data store in your subscription and resource group. Memory, knowledge, threads, and session insights live in your chosen region. Nothing is used to train the model provider. Encryption at rest, TLS 1.2 in transit, Azure RBAC, managed identity, customer-managed keys all apply. Identity boundary you already know. The agent uses standard Azure managed identity. Grant the identity RBAC on any cross-subscription resource it needs to reach. Least privilege still applies. Region availability. At session time, agents can be deployed in EastUS2, Sweden Central, and Australia East. The list is updating roughly monthly. Canada is coming. An agent in one region can act on resources globally, but if you have data residency rules, deploy the agent inside the same jurisdiction. Private endpoints today. If your Log Analytics Workspace or databases are fully locked down behind private endpoints with public access disabled, the agent currently needs a VNet-integrated Azure Function as a proxy. Microsoft is actively working on injecting agents directly into private networks. Memory is the multiplier. A principal engineer is more valuable than a junior engineer because of pattern recognition. SRE Agent captures that pattern recognition for the whole team, every time it investigates. Getting Started The pattern is simple, and Lee summarized it cleanly: you teach the tool, you make the connections, and it works for you. Provision the agent. Go to sre.azure.com or the Azure portal, pick a subscription and resource group, pick a region, and stand it up. Takes a few minutes. Onboard it like a new engineer. Tell it about your team, your workloads, and your procedures. Upload runbooks, troubleshooting guides, wikis, and architecture docs to the knowledge base. If you do not have these documents, ask the agent to draft them for you. Connect your observability stack. Azure Monitor, Log Analytics, App Insights are wired in by default. Add third-party tools through MCP connectors. Wire in your incident platform. Azure Monitor, PagerDuty, or ServiceNow. One agent per platform. Grant code access. Connect your GitHub or Azure DevOps repositories so the agent can reason over application code, propose fixes, and open pull requests. Pick your run mode. Start in interactive mode while you build trust. Move to approval-gated reactive mode. Graduate to autonomous mode on safe operations once you have the audit trail you trust. Resources Azure SRE Agent documentation on Microsoft Learn Azure SRE Agent product docs Get Started guide Automate incident response Official Microsoft SRE Agent GitHub repository (issues, labs, resources) Watch the Rest of the Summit If you found this useful, the rest of the Azure Infra Summit 2026 is packed with sessions on identity, AKS, deployment, storage, networking, and resiliency. Grab the full playlist here and binge what is relevant to your stack:Microsoft Azure Infra Summit 2026 Big thanks to Lee Oommen for walking us through this. The 3 a.m. pager scenario is something every one of us has lived, and seeing an agent take the first hour of that incident off your plate is a tangible win. Cheers! Pierre Roman
At-Scale Failure Reporting for Azure Update Manager
Introduction Azure Update Manager simplifies patching across Azure virtual machines and Azure Arc-enabled servers by providing a centralized platform for patch assessment and installation. However, as environments scale, a key challenge emerges—efficiently identifying and troubleshooting patch failures across large fleets of machines. While Azure Update Manager surfaces detailed error messages in the Azure portal, this information is typically available only at an individual machine level. In enterprise environments managing hundreds or thousands of systems, drilling into each VM to find error details quickly becomes impractical. In this article, we walk through a real-world use case and demonstrate how to leverage Azure Resource Graph (ARG) to extract failed machines along with their error details for a specific maintenance run—using a single query. The Challenge: Scaling Patch Failure Visibility In a large enterprise deployment, Azure Update Manager was configured to manage patching across: Windows and Linux virtual machines Azure cloud VMs and Arc-enabled on‑premises servers Multiple regions and subscriptions While patching operations were largely successful, a subset of machines experienced failures. The key challenges faced by the operations team were: Error messages were visible only by drilling into each failed VM in the portal No built‑in way to aggregate failures across all machines Lack of a simple mechanism to export: Failed VMs Error codes Error messages The team needed a scalable, query‑driven approach to analyze failures across an entire maintenance run. Key Insight: Where Azure Update Manager Stores Data Azure Update Manager does not rely on Log Analytics to store operational results. Instead: Patch assessment and installation results are stored in Azure Resource Graph Azure Resource Graph acts as a centralized, queryable store for update operations This design enables powerful querying without requiring additional ingestion, configuration, or cost overhead. Understanding Maintenance Runs and Correlation IDs Each Azure Update Manager maintenance run generates a unique identifier: properties.correlationId represents the maintenance (schedule) run ID All machines involved in the same patch cycle share this ID This allows all machines within a single patch execution to be correlated and queried collectively. The Solution: Query Failed VMs with Error Messages Azure Resource Graph allows querying failures at scale using the maintenanceresources dataset. Core Query (Kusto Query Language) 1 maintenanceresources 2 | where type =~ "microsoft.maintenance/applyupdates" 3 | where tostring(properties.correlationId) contains "<YourMaintenanceRunID>" 4 | where tostring(properties.status) =~ "Failed" 5 | project properties.resourceId, properties.errorCode, properties.errorMessage What This Query Delivers All machines that failed in a specific maintenance run Error codes for troubleshooting Full error messages that are otherwise visible only in the Azure portal Note: Property names for error information can vary by environment. Validate available fields using Azure Resource Graph Explorer and adjust the project clause if required. Sample Output (Conceptual) Resource ID Error Code Error Message vm-01 0x80244007 Windows Update API failed vm-02 0x80072f8f Connectivity issue vm-03 1C WSUS configuration issue Advanced Scenario: Automatically Detecting the Latest Failed Maintenance Run In real-world scenarios, you may not always know the maintenance run ID. The following query dynamically identifies the most recent maintenance run that had failures, and then retrieves all failed machines from that run. 1 // Step 1: Identify the latest maintenance run ID with failures 2 let lastFailedRun = toscalar( 3 maintenanceresources 4 | extend runId = extract(@"applyupdates/(\d+)$", 1, properties.correlationId) 5 | where type =~ "microsoft.maintenance/applyupdates" 6 | where tostring(properties.status) =~ "Failed" 7 | order by tostring(properties.startDateTime) desc 8 | take 1 9 | project runId 10 ); 11 // Step 2: Query all failed VMs from that run 12 maintenanceresources 13 | where type =~ "microsoft.maintenance/applyupdates" 14 | where tostring(properties.correlationId) contains lastFailedRun 15 | where tostring(properties.status) =~ "Failed" 16 | project properties.resourceId, properties.errorCode, properties.errorMessage This approach is ideal for automation, scheduled reporting, and dashboard scenarios. Why This Approach Matters Operational Efficiency Eliminates manual portal navigation Provides consolidated failure insights in seconds Scalability Works across large, distributed environments Supports both Azure and hybrid (Arc‑enabled) machines Automation Ready Can be integrated into scripts, dashboards, and reporting pipelines Enables proactive monitoring and alerting scenarios Best Practices for Enterprise Patch Reporting To maximize the value of this approach: Capture and track maintenance run IDs Use Azure Resource Graph as the primary reporting layer Build reusable queries for different patch scenarios Export reports for compliance and auditing Correlate failures with root‑cause trends over time Conclusion As organizations scale patching operations with Azure Update Manager, visibility, speed, and automation become essential. While the Azure portal is effective for per‑machine troubleshooting, it is not optimized for fleet‑level analysis. Azure Resource Graph fills this gap by enabling a shift from manual troubleshooting to automated, query‑driven failure analysis at scale. By adopting this approach, teams can significantly improve operational efficiency, reduce mean time to resolution, and build a more mature patch management strategy. Final takeaway: Don’t rely only on the portal Leverage Azure Resource Graph to operationalize patch insights at enterprise scale References Azure Update Manager – Query resources with Azure Resource Graph https://learn.microsoft.com/azure/update-manager/query-logs Azure Update Manager – Troubleshooting guide https://learn.microsoft.com/azure/update-manager/troubleshoot Sample Azure Resource Graph queries for Azure Update Manager https://github.com/MicrosoftDocs/azure-docs/blob/main/articles/update-manager/sample-query-logs.md
