Connect Azure SRE Agent to ServiceNow: End-to-End Incident Response
🎯 What You'll Achieve In this tutorial, you'll: Connect Azure SRE Agent to ServiceNow as your incident management platform Create a test incident in ServiceNow Watch the AI agent automatically pick up, investigate, and resolve the incident See the agent write triage findings and resolution notes back to ServiceNow Time to complete: ~10 minutes 🎬 The End Result Before we dive in, here's what the end result looks like: ServiceNow Incident - Resolved by Azure SRE Agent The Azure SRE Agent: ✅ Detected the incident from ServiceNow ✅ Acknowledged and began triage automatically ✅ Investigated AKS cluster memory utilization ✅ Documented findings in work notes ✅ Resolved the incident with detailed root cause analysis 📋 Prerequisites A ServiceNow instance (Developer, PDI, or Enterprise) Administrator access to ServiceNow An Azure SRE Agent deployed in your Azure subscription 💡 Don't have a ServiceNow instance? Get a free Personal Developer Instance (PDI) at developer.servicenow.com 🔧 Step 1: Gather Your ServiceNow Credentials To connect Azure SRE Agent to ServiceNow, you need three pieces of information: Component Where to Find It ServiceNow Endpoint Browser address bar when logged into ServiceNow (format: https://your-instance.service-now.com ) Username Click your profile avatar → Profile → User ID Password Your ServiceNow login password Finding Your Instance URL Your ServiceNow instance URL is visible in your browser's address bar when logged in: https://{your-instance-name}.service-now.com Finding Your Username Click your profile avatar in the top-right corner of ServiceNow Click Profile Your User ID is your username ⚙️ Step 2: Connect SRE Agent to ServiceNow Navigate to Your SRE Agent Open the Azure Portal Search for "Azure SRE Agent" in the search bar Click on Azure SRE Agent (Preview) in the results Select your agent from the list Configure the Incident Platform In the left navigation, expand Settings Click Incident platform Click the Incident platform dropdown Select ServiceNow Here's what the ServiceNow configuration form looks like: Enter Your ServiceNow Credentials Field Value ServiceNow endpoint Your ServiceNow instance URL (from Step 1) Username Your ServiceNow username (from Step 1) Password Your ServiceNow password Quickstart response plan ✓ Enable this for automatic investigation Save and Verify Click the Save button Wait for validation to complete Look for: "ServiceNow is connected." with a green checkmark 🚨 Step 3: Create a Test Incident in ServiceNow Now let's test the integration by creating an incident in ServiceNow. Navigate to Create Incident In ServiceNow, click All in the left navigation Search for "Incident" Click Incident → Create New Fill in the Incident Details Field Value Caller Select any user (e.g., System Administrator) Short description [SRE Agent Test] AKS Cluster memory pressure detected in production environment Impact 2 - Medium Submit the Incident Click Submit to create the incident. Note the incident number that's assigned. 🤖 Step 4: Watch SRE Agent Investigate Check the SRE Agent Portal Return to the Azure Portal Open your SRE Agent Click Activities → Incidents Within seconds, you should see your ServiceNow incident appear! Observe the Autonomous Investigation Click on the incident to see the SRE Agent's investigation in action: The agent automatically: 🔔 Acknowledged the incident 📋 Created a triage plan with clear steps 🔍 Identified AKS clusters in your subscription 📊 Validated memory utilization metrics ✅ Resolved the incident with findings 📝 Step 5: Review Resolution in ServiceNow Check the ServiceNow Incident Return to ServiceNow and open your incident. You'll see: State: Changed to Resolved Activity Stream: Multiple work notes from the agent Resolution notes: Detailed findings Resolution Notes The agent writes comprehensive resolution notes including: Timestamp of resolution Root cause analysis Validation steps performed Fix applied (if any) 🚀 Next Steps Create custom response plans to customize how the agent responds to different incident types Configure alert routing to route specific Azure Monitor alerts to the agent Explore the Azure SRE Agent documentation for more features Share your experiences, learnings, and questions with other early adopters. Start a discussion in our Community Hub
Proactive Cloud Ops with SRE Agent: Scheduled Checks for Cloud Optimization
The Cloud Optimization Challenge Your cloud environment is always changing: New features ship weekly Traffic patterns shift seasonally Costs creep up quietly Security best practices evolve Teams spin up resources and forget them It's Monday morning. You open the Azure portal. Everything looks... fine. But "fine" isn't great. That VM has been at 8% CPU for weeks. A Key Vault secret expires in 12 days. Nothing's broken. But security is drifting, costs are creeping, and capacity gaps are growing silently. The question isn't "is something broken?" it's "could this be better?" Four Pillars of Cloud Optimization Pillar What Teams Want The Challenge Security Stay compliant, reduce risk Config drift, legacy settings, expiring creds Cost Spend efficiently, justify budget Hard to spot waste across 100s of resources Performance Meet SLOs, handle growth Know when to scale before demand hits Availability Maximize uptime, build resilience Hidden dependencies, single points of failure Most teams check these sometimes. SRE Agent checks them continuously. Enter SRE Agent + Scheduled tasks SRE Agent can pull data from Azure Monitor, resource configurations, metrics, logs, traces, errors, cost data and analyze it on a schedule. If you use tools outside Azure (Datadog, PagerDuty, Splunk), you can connect those via MCP servers so the agent sees your full observability stack. My setup uses Azure-native sources. Here's how I wired it up. How I Set It Up: Step by Step Step 1: Create SRE Agent with Subscription Access I created an SRE Agent without attaching it to any specific resource group. Instead, I gave it Reader access at the subscription level. This lets the agent scan across all my resource groups for optimization opportunities. No resource group configuration needed. The agent builds a knowledge graph of everything VMs, storage accounts, Key Vaults, NSGs, web apps across the subscription. Step 2: Create and Upload My Organization Practices I created an org-practices.md file that defines what "good" looks like for my team: I uploaded this to SRE Agent's knowledge base. Now the agent knows our bar, not just Azure defaults. 👉 See my full org-practices.md Source repos for this demo: security-demoapp - App with intentional security misconfigurations costoptimizationapp - App with cost optimization opportunities Step 3: Connect to Teams Channel I connected SRE Agent to my team's Teams channel so findings land where we already work. Critical findings get immediate notifications. Warnings go into a daily digest. No more logging into separate dashboards. The insights come to us. Step 4: Connect Resource Groups to GitHub Repos Add the two resource groups to the SRE Agent and link the apps to their corresponding GitHub repos: Resource Group GitHub Repository rg-security-opt-demo security-demoapp rg-cost-opt-sreademo costoptimizationapp This enables the agent to create GitHub issues for findings linking violations directly to the repo responsible for that infrastructure. Step 5: Test with Prompts Before setting up automation, I tested the agent with manual prompts to make sure it was finding the right issues. The agent ran the checks, compared against my org-practices.md, and identified the issues. Security Check: Scan resource group "rg-security-opt-demo" for any violations of our security practices defined in org-practices.md in your knowledge base. list violations with severity and remediation steps. Make sure to check against all critical requirements and send message in teams channel with your findings and create an issue in the github repo https://github.com/dm-chelupati/security-demoapp.git Cost Check: Scan resource group "rg-cost-opt-sreademo" for any violations of our costpractices defined in org-practices.md in your knowledge base. list violations with severity and remediation steps. Make sure to check against all critical requirements and send message in teams channel with your findings and create an issue in the github repo https://github.com/dm-chelupati/costoptimizationapp.git Step 6: Check Output via GitHub Issues After running prompts, I checked GitHub. The agent had created issues. Each issue has the root cause, impact, and fix ready for the team to action or for Coding Agent to pick up and create a PR. 👉 See the actual issues created: Security findings issue Cost findings issue Step 7: Set Up Scheduled Triggers This is where it gets powerful. I configured recurring schedules: Weekly Security Check (Wednesdays 8 AM): Create a scheduled trigger that performs security practices checks against the org practices in knowledge base org-practices.md, creates github issue and send teams message on a weekly basis Wednesdays at 8 am UTC Weekly Cost Review (Mondays 8 AM): Create a scheduled trigger that performs cost practices checks against the org practices in knowledge base org-practices.md, creates github issue and send teams message on a weekly basis on Mondays at 8 am UTC Now optimization runs automatically. Every week, fresh findings land in GitHub Issues and Teams. Why Context Makes the SRE Agent Powerful Think about hiring a new SRE. They're excellent at their craft—they know Kubernetes, networking, Azure inside out. But on day one, they can't solve problems in your environment yet. Why? They don't have context: What are your SLOs? What's "acceptable" latency for your app? When do you rotate secrets? Monthly? Quarterly? Before each release? Which resources are production-critical vs. dev experiments? What's your tagging policy? Who owns what? How do you deploy? GitOps? Pipelines? Manual approvals? A great engineer becomes your great engineer once they learn how your team operates. SRE Agent works the same way. Out of the box, it knows Azure resource types, networking, best practices. But it doesn't know your bar. Is 20% CPU utilization acceptable or wasteful? Should secrets expire in 30 days or 90? Are public endpoints ever okay, or never? The more context you give the agent, your SLOs, your runbooks, your policies, the more it reasons like a team member who understands your environment, not just Azure in general. That's why Step 2 matters so much. When I uploaded our standards, the agent stopped checking generic Azure best practices and started checking our best practices. Bring your existing knowledge: You don't have to start from scratch. If your team's documentation already lives in Atlassian Confluence, SharePoint, or other tools, you can connect those via MCP servers. The agent pulls context from where your team already works, no need to duplicate content. Why This Matters Before this setup, optimization was a quarterly thing. Now it happens automatically: Before After Check security when audit requests it Daily automated posture check Find waste when finance complains Weekly savings report in Teams Discover capacity issues during incidents Scheduled headroom analysis Expire credentials and debug at 2 AM 30-day warning with exact secret names Optimization isn't a project anymore. It's a practice. Try It Yourself Create an SRE Agent with access to your subscription Upload your team's standards (security policies, cost thresholds, tagging rules) Set up a scheduled trigger, start with a daily security check Watch the first report land in Teams See what you've been missing while everything looked "fine." Learn More Azure SRE Agent documentation Azure SRE Agent blogs Azure SRE Agent community Azure SRE Agent home page Azure SRE Agent pricing Azure SRE Agent is currently in preview. Get StartedContext Engineering Lessons from Building Azure SRE Agent
We started with 100+ tools and 50+ specialized agents. We ended with 5 core tools and a handful of generalists. The agent got more reliable, not less. Every context decision is a tradeoff: latency vs autonomy, evidence-building vs speed, oversight - and the cost of being wrong. This post is a practical map of those knobs and how we adjusted them for SRE Agent.Find the Alerts You Didn't Know You Were Missing with Azure SRE Agent
I had 6 alert rules. CPU. Memory. Pod restarts. Container errors. OOMKilled. Job failures. I thought I was covered. Then my app went down. I kept refreshing the Azure portal, waiting for an alert. Nothing. That's when it hit me: my alerts were working perfectly. They just weren't designed for this failure mode. Sound familiar? The Problem Every Developer Knows If you're a developer or DevOps engineer, you've been here: a customer reports an issue, you scramble to check your monitoring, and then you realize you don't have the right alerts set up. By the time you find out, it's already too late. You set up what seems like reasonable alerting and assume you're covered. But real-world failures are sneaky. They slip through the cracks of your carefully planned thresholds. My Setup: AKS with Redis I love to vibe code apps using GitHub Copilot Agent mode with Claude Opus 4.5. It's fast, it understands context, and it lets me focus on building rather than boilerplate. For this project, I built a simple journal entry app: AKS cluster hosting the web API Azure Cache for Redis storing journal data Azure Monitor alerts for CPU, memory, pod restarts, container errors, OOMKilled, and job failures Seemed solid. What could go wrong? The Scenario: Redis Password Rotation Here's something that happens constantly in enterprise environments: the security team rotates passwords. It's best practice. It's in the compliance checklist. And it breaks things when apps don't pick up the new credentials. I simulated exactly this. The pods came back up. But they couldn't connect to Redis (as expected). The readiness probes started failing. The LoadBalancer had no healthy backends. The endpoint timed out. And not a single alert fired. Using SRE Agent to Find the Alert Gaps Instead of manually auditing every alert rule and trying to figure out what I missed, I turned to Azure SRE Agent. I asked it a simple question: "My endpoint is timing out. What alerts do I have, and why didn't any of them fire?" Within minutes, it had diagnosed the problem. Here's what it found: My Existing Alerts Why They Didn't Fire High CPU/Memory No resource pressure,just auth failures Pod Restarts Pods weren't restarting, just unhealthy Container Errors App logs weren't being written OOMKilled No memory issues Job Failures No K8s jobs involved The gaps SRE Agent identified: ❌ No synthetic URL availability test ❌ No readiness/liveness probe failure alerts ❌ No "pods not ready" alerts scoped to my namespace ❌ No Redis connection error detection ❌ No ingress 5xx/timeout spike alerts ❌ No per-pod resource alerts (only node-level) SRE Agent didn't just tell me what was wrong, it created a GitHub issue with : KQL queries to detect each failure type Bicep code snippets for new alert rules Remediation suggestions for the app code Exact file paths in my repo to update Check it out: GitHub Issue How I Built It: Step by Step Let me walk you through exactly how I set this up inside SRE Agent. Step 1: Create an SRE Agent I created a new SRE Agent in the Azure portal. Since this workflow analyzes alerts across my subscription (not just one resource group), I didn't configure any specific resource groups. Instead, I gave the agent's managed identity Reader permissions on my entire subscription. This lets it discover resources, list alert rules, and query Log Analytics across all my resource groups. Step 2: Connect GitHub to SRE Agent via MCP I added a GitHub MCP server to give the agent access to my source code repository.MCP (Model Context Protocol) lets you bring any API into the agent. If your tool has an API, you can connect it. I use GitHub for both source code and tracking dev tickets, but you can connect to wherever your code lives (GitLab, Azure DevOps) or your ticketing system (Jira, ServiceNow, PagerDuty). Step 3: Create a Subagent inside SRE Agent for managing Azure Monitor Alerts I created a focused subagent with a specific job and only the tools it needs: Azure Monitor Alerts Expert Prompt: " You are expert in managing operations related to azure monitor alerts on azure resources including discovering alert rules configured on azure resources, creating new alert rules (with user approval and authorization only), processing the alerts fired on azure resources and identifying gaps in the alert rules. You can get the resource details from azure monitor alert if triggered via alert. If not, you need to ask user for the specific resource to perform analysis on. You can use az cli tool to diagnose logs, check the app health metrics. You must use the app code and infra code (bicep files) files you have access to in the github repo <insert your repo> to further understand the possible diagnoses and suggest remediations. Once analysis is done, you must create a github issue with details of analysis and suggested remediation to the source code files in the same repo." Tools enabled: az cli – List resources, alert rules, action groups Log Analytics workspace querying – Run KQL queries for diagnostics GitHub MCP – Search repositories, read file contents, create issues Step 4: Ask the Subagent About Alert Gaps I gave the agent context and asked a simple question: "@AzureAlertExpert: My API endpoint http://132.196.167.102/api/journals/john is timing out. What alerts do I have configured in rg-aks-journal, and why didn't any of them fire? The agent did the analysis autonomously and summarized findings with suggestions to add new alert rules in a GitHub issue. Here's the agentic workflow to perform azure monitor alert operations Why This Matters Faster response times. Issues get diagnosed in minutes, not hours of manual investigation. Consistent analysis. No more "I thought we had an alert for that" moments. The agent systematically checks what's covered and what's not. Proactive coverage. You don't have to wait for an incident to find gaps. Ask the agent to review your alerts before something breaks. The Bottom Line Your alerts have gaps. You just don't know it until something slips through. I had 6 alert rules and still missed a basic failure. My pods weren't restarting, they were just unhealthy. My CPU wasn't spiking, the app was just returning errors. None of my alerts were designed for this. You don't need to audit every alert rule manually. Give SRE Agent your environment, describe the failure, and let it tell you what's missing. Stop discovering alert gaps from customer complaints. Start finding them before they matter. A Few Tips Give the agent Reader access at subscription level so it can discover all resources Use a focused subagent prompt, don't try to do everything in one agent Test your MCP connections before running workflows What Alert Gaps Have Burned You? What's the alert you wish you had set up before an incident? Credential rotation? Certificate expiry? DNS failures? Let us know in the comments.From Vibe Coding to Working App: How SRE Agent Completes the Developer Loop
The Most Common Challenge in Modern Cloud Apps There's a category of bugs that drive engineers crazy: multi-layer infrastructure issues. Your app deploys successfully. Every Azure resource shows "Succeeded." But the app fails at runtime with a vague error like Login failed for user ''. Where do you even start? You're checking the Web App, the SQL Server, the VNet, the private endpoint, the DNS zone, the identity configuration... and each one looks fine in isolation. The problem is how they connect and that's invisible in the portal. Networking issues are especially brutal. The error says "Login failed" but the actual causes could be DNS, firewall, identity, or all three. The symptom and the root causes are in completely different resources. Without deep Azure networking knowledge, you're just clicking around hoping something jumps out. Now imagine you vibe coded the infrastructure. You used AI to generate the Bicep, deployed it, and moved on. When it breaks, you're debugging code you didn't write, configuring resources you don't fully understand. This is where I wanted AI to help not just to build, but to debug. Enter SRE Agent + Coding Agent Here's what I used: Layer Tool Purpose Build VS Code Copilot Agent Mode + Claude Opus Generate code, Bicep, deploy Debug Azure SRE Agent Diagnose infrastructure issues and create developer issue with suggested fixes in source code (app code and IaC) Fix GitHub Coding Agent Create PRs with code and IaC fix from Github issue created by SRE Agent Copilot builds. SRE Agent debugs. Coding Agent fixes. What I Built I used VS Code Copilot in Agent Mode with Claude Opus to create a .NET 8 Web App connected to Azure SQL via private endpoint: Private networking (no public exposure) Entra-only authentication Managed identity (no secrets) Deployed with azd up. All green. Then I tested the health endpoint: $ curl https://app-tsdvdfdwo77hc.azurewebsites.net/health/sql {"status":"unhealthy","error":"Login failed for user ''.","errorType":"SqlException"} Deployment succeeded. App failed. One error. How I Fixed It: Step by Step Step 1: Create SRE Agent with Azure Access I created an SRE Agent with read access to my Azure subscription. You can scope it to specific resource groups. The agent builds a knowledge graph of your resources and their dependencies visible in the Resource Mapping view below. Step 2: Connect GitHub to SRE Agent using GitHub MCP server I connected the GitHub MCP server so the agent could read my repository and create issues. Step 3: Create Sub Agent to analyze source code I created a sub-agent for analyzing source code using GitHub mcp tools. this lets SRE Agent understand not just Azure resources, but also the Bicep and source code files that created them. "you are expert in analyzing source code (bicep and app code) from github repos" Step 4: Invoke Sub-Agent to Analyze the Error In the SRE Agent chat, I invoked the sub-agent to diagnose the error I received from my app end point. It correlated the runtime error with the infrastructure configuration Step 5: Watch the SRE Agent Think and Reason SRE Agent analyzed the error by tracing code in Program.cs, Bicep configurations, and Azure resource relationships Web App, SQL Server, VNet, private endpoint, DNS zone, and managed identity. Its reasoning process worked through each layer, eliminating possibilities one by one until it identified the root causes. Step 6: Agent Creates GitHub Issue Based on its analysis, SRE Agent summarized the root causes and suggested fixes in a GitHub issue: Root Causes: Private DNS Zone missing VNet link Managed identity not created as SQL user Suggested Fixes: Add virtualNetworkLinks resource to Bicep Add SQL setup script to create user with db_datareader and db_datawriter roles Step 7: Merge the PR from Coding Agent Assign the Github issue to Coding Agent which then creates a PR with the fixes. I just reviewed the fix. It made sense and I merged it. Redeployed with azd up, ran the SQL script: curl -s https://app-tsdvdfdwo77hc.azurewebsites.net/health/sql | jq . { "status": "healthy", "database": "tododb", "server": "tcp:sql-tsdvdfdwo77hc.database.windows.net,1433", "message": "Successfully connected to SQL Server" } 🎉 From error to fix in minutes without manually debugging a single Azure resource. Why This Matters If you're a developer building and deploying apps to Azure, SRE Agent changes how you work: You don't need to be a networking expert. SRE Agent understands the relationships between Azure resources private endpoints, DNS zones, VNet links, managed identities. It connects dots you didn't know existed. You don't need to guess. Instead of clicking through the portal hoping something looks wrong, the agent systematically eliminates possibilities like a senior engineer would. You don't break your workflow. SRE Agent suggests fixes in your Bicep and source code not portal changes. Everything stays version controlled. Deployed through pipelines. No hot fixes at 2 AM. You close the loop. AI helps you build fast. Now AI helps you debug fast too. Try It Yourself Do you vibe code your app, your infrastructure, or both? How do you debug when things break? Here's a challenge: Vibe code a todo app with a Web App, VNet, private endpoint, and SQL database. "Forget" to link the DNS zone to the VNet. Deploy it. Watch it fail. Then point SRE Agent at it and see how it identifies the root cause, creates a GitHub issue with the fix, and hands it off to Coding Agent for a PR. Share your experience. I'd love to hear how it goes. Learn More Azure SRE Agent documentation Azure SRE Agent blogs Azure SRE Agent community Azure SRE Agent home page Azure SRE Agent pricingExtend SRE Agent with MCP: Build an Agentic Workflow to Triage Customer Issues
Your inbox is full. GitHub issues piling up. "App not working." "How do I configure alerts?" "Please add dark mode." You open each one, figure out what it is, ask for more info, add labels, route to the right team. An hour later, you're still sorting issues. Sound familiar? The Triage Tax Every L1 support engineer, PM, and on-call developer who's handled customer issues knows this pain. When tickets come in, you're not solving problems, you're sorting them. Read the issue. Is it a bug or a question? Check the docs. Does this feature exist? Ask for more info. Wait two days. Re-triage. Add labels. Route to engineering. It's tedious. It requires judgment, you need to understand the product, know what info is needed, check documentation. And honestly? It's work that nobody volunteers for but someone has to do. In large organizations, it gets even more complex. The issue doesn't just need to be triaged, it needs to be routed to the right engineering team. Is this an auth issue? Frontend? Backend? Infrastructure? A wrong routing decision means delays, re-assignments, and frustrated customers. What if an AI agent could do this for you? Enter Azure SRE Agent + MCP Here's what I built: I gave SRE Agent access to my GitHub and PagerDuty accounts via MCP, uploaded my triage rubric as a markdown file, and set it to run twice a day. No more reading every ticket manually. No more asking the same "please provide more info" questions. No more morning triage sessions. What My Setup Looks Like My app's customer issues come in through GitHub. My team uses PagerDuty to track bugs and incidents. So I connected both via MCP to the SRE Agent. I also uploaded my triage logic as a .md file on how to classify issues, what info is required for each category, which labels to use, which team handles what. And since I didn't want to run this workflow manually, I set up a scheduled task to trigger it twice a day. Now it just runs. I verify its work if I want to. What the Agent Does Fetches all open, unlabeled GitHub issues Reads each issue and classifies it (bug, doc question, feature request) Checks if required info is present Posts a comment asking for details if needed, or acknowledges the issue Adds appropriate labels Creates a PagerDuty incident for bugs ready for engineering Moves to the next issue How I Built It: Step by Step Let me walk you through exactly how I set this up inside SRE Agent. Step 1: Create an SRE Agent I created a new SRE Agent in the Azure portal. Since this workflow triages GitHub issues and not Azure resources, I didn't need to configure any Azure resource groups or subscriptions. Just an agent. Step 2: Connect MCP Servers I added two MCP servers to give the agent access to my tools: GitHub MCP– Fetch issues, post comments, add labels PagerDuty MCP – Create incidents for bugs that need dev team's attention MCP (Model Context Protocol) lets you bring any API into the agent. If your tool has an API, you can connect it. Step 3: Create Subagents I created two focused subagents, each with a specific job and only the tools it needs: GitHub Issue Triager "You are expert in triaging GitHub issues, classifying them into categories such as user needs to supply additional information, bug, documentation question, or feature request. Use the knowledge base to search for the right document that helps you with performing this triaging. Perform all actions autonomously without waiting for user input. Hand off to Incident Creator for the issues you classified as bugs." Tools: GitHub MCP (issues, labels, comments) Incident Creator Here "You are expert in managing incidents in PagerDuty, listing services, incidents, creating incidents with all details. Once done, hand off back to GitHub Issue Triager." Tools: PagerDuty MCP (services, incidents) The handoff between them creates a workflow. They collaborate without human involvement. Step 4: Add Your Knowledge I uploaded my triage logic as a .md file to the agent's knowledge base. This is my rubric - my mental model for how to triage issues: How do I classify bugs vs. doc questions vs. feature requests? What info is required for each category? What labels do I use? When should an incident be created? Which team handles which type of issue? I wrote it down the way I'd explain it to a new teammate. The agent searches and follows it. Step 5: Add a Scheduled Task I didn't want to trigger this workflow manually every time. SRE Agent supports scheduled tasks, workflows that run automatically on a cadence. I set up a trigger to run twice a day: morning and evening. Now the workflow is fully automated. Here is the end to end automated agentic workflow to triage customer tickets. Why MCP Matters Every team uses different tools. Maybe your customer issues live in Zendesk, incidents go to ServiceNow and you use Jira or Azure DevOps. SRE Agent doesn't lock you in. With MCP, you connect to whatever tools you already use. The agent orchestrates across them. That's the extensibility model: your tools, your workflow, orchestrated by the agent. The Result Before: 2 hours every morning sorting tickets. After: By the time anyone logs in, issues are labeled, missing-info requests are posted, urgent bugs have incidents, and feature requests are acknowledged. Your team can finally focus on the complex stuff not sorting tickets. Why This Matters Faster response times. Issues get acknowledged in minutes, not days. Consistent classification. No "this should have been a P1" moments. No tickets bouncing between teams. Happier customers. They get a response immediately even if it's just "we're looking into it." Focus on what matters. Your team should be solving problems, not sorting them. The Bottom Line Triage isn't the job, it's the tax on the job. It quietly eats the hours your team could spend building, debugging, and shipping. You don't need to build a custom triage bot. You don't need to wire up webhooks and write glue code. You give the SRE agent your tools, your logic, and a schedule and it handles the sorting. Use GitHub? Connect GitHub. Use Zendesk? Connect Zendesk. PagerDuty, ServiceNow, Jira - whatever your team runs on, the agent meets you there. Stop sorting tickets. Start shipping. A Few Tips Test MCP endpoints before configuring them in the SRE agent Give each subagent only the tools it needs, don't enable everything Start read-only until you trust the classification, then enable comments Do You Still Want to Triage Issues Manually? What tools does your team use to track customer-reported issues and incidents? Let us know in the comments, we'd love to hear how you'd use this workflow with your stack. Is triage your most toilsome workflow or is there something even worse eating your team's time? Let us know in the comments.
Fix It Before They Feel It: Higher Reliability with Proactive Mitigation
What if your infrastructure could detect performance issues and fix them automatically—before your users even notice? This blog brings that vision to life using Azure SRE Agent, an AI-powered autonomous agent that monitors, detects, and remediates production issues in real-time. 💡 The magic: Zero human intervention required. The agent handles detection, diagnosis, remediation, and reporting—all autonomously. 📺 Watch the Demo This content was presented at .NET Day 2025. Watch the full session to see Azure SRE Agent in action: 🎬 Fix it before they feel it - .NET Day 2025 🎯 What You'll See in This Demo Watch as we intentionally deploy "bad" code to production and observe how the SRE Agent: Detects the degradation — Compares live response times against learned baselines Takes autonomous action — Executes a slot swap to roll back to healthy code Communicates the incident — Posts to Teams and creates a GitHub issue Generates reports — Summarizes deployment metrics for stakeholders 🚀 Key Capabilities Capability What It Shows Proactive Baseline Learning Agent learns normal response times and stores them in a knowledge base Real-time Anomaly Detection Instant comparison of current vs. baseline metrics Autonomous Remediation Agent executes Azure CLI commands to swap slots without human approval Cross-platform Communication Automatic Teams posts and GitHub issue creation Incident Reporting End-of-day email summaries with deployment health metrics Architecture Overview The solution uses Azure SRE Agent with three specialized sub-agents working together: Components Application Layer: .NET 9 Web API running on Azure App Service Application Insights for telemetry collection Azure Monitor Alerts for incident triggers Azure SRE Agent: AvgResponseTime Sub-Agent: Captures baseline metrics every 15 minutes, stores in Knowledge Store DeploymentHealthCheck Sub-Agent: Triggered by deployment alerts, compares metrics to baseline, auto-remediates DeploymentReporter Sub-Agent: Generates daily summary emails from Teams activity External Integrations: GitHub (issue creation, semantic code search, Copilot assignment) Microsoft Teams (deployment summaries) Outlook (summary reports) What the SRE Agent Does When a deployment occurs, the SRE Agent autonomously performs the following actions: 1. Health Check (DeploymentHealthCheck Sub-Agent) When a slot swap alert fires, the agent: Queries App Insights for current response times Retrieves the baseline from the Knowledge Store Compares current performance against baseline If degradation > 20%: Executes rollback and creates GitHub issue If healthy: Posts confirmation to Teams Healthy Deployment - No Action Needed (Teams Post): The agent confirms the deployment is healthy — response time (22ms) is 80% faster than baseline (116ms). Degraded Deployment - Automatic Rollback (Teams Post): The agent detects +332% latency regression (212ms vs 116ms baseline), executes a slot swap to rollback, and creates GitHub Issue. 2. Daily Summary (DeploymentReporter Sub-Agent) Every 24 hours, the reporter agent: Reads all Teams deployment posts from the last 24 hours Aggregates deployment metrics Sends an executive summary email Daily Summary (Outlook Email): The daily report shows 9 deployments, 6 healthy, 3 rollbacks, and 3 GitHub issues created — complete with response time details and issue links. Demo Flow View Step-by-Step Instructions → Step Action Step 1 Deploy Infrastructure + Applications Step 2 Create Sub-Agents, Triggers & Schedules Step 3 Swap bad code, watch agent remediate Setting Up the Demo Prerequisites Azure subscription with Contributor access Azure CLI installed and logged in (az login) .NET 9.0 SDK PowerShell 7.0+ Step 1: Deploy Infrastructure cd scripts .\1-setup-demo.ps1 -ResourceGroupName "sre-demo-rg" -AppServiceName "sre-demo-app-12345" This script will: Prompt for Azure subscription selection Deploy Azure infrastructure (App Service, App Insights, Alerts) Build and deploy healthy code to production Build and deploy problematic code to staging Full Setup Instructions → Step 2: Configure Azure SRE Agent Navigate to Azure SRE Agents Portal Sub Agent builder tab and create three sub-agents: Sub-Agent Purpose Tools Used AvgResponseTime Captures baseline response time metrics QueryAppInsightsByAppId, UploadKnowledgeDocument DeploymentHealthCheck Detects degradation and executes remediation SearchMemory, QueryAppInsights, PostTeamsMessage, CreateGithubIssue, Az CLI commands DeploymentReporter Generates deployment summary reports GetTeamsMessages, SendOutlookEmail Creating Each Sub-Agent In the Github links below you can find gif images that capture the creation flow AvgResponseTime + Baseline Task: Detailed Instructions → DeploymentHealthCheck + Swap Alert: Detailed Instructions → DeploymentReporter + Reporter Task: Detailed Instructions → Step 3: Run the Demo .\2-run-demo.ps1 This triggers the following flow: Slot Swap Occurs (demo script) ▼ Activity Log Alert Fires ▼ Incident Trigger Activated ▼ DeploymentHealthCheck Agent Runs ─ Queries current response time from App Insights ─ Retrieves baseline from knowledge store ─ Compares (if >20% degradation) ─ Executes: az webapp deployment slot swap ─ Creates GitHub issue (if degraded) ─ Posts to Teams channel Full Demo Instructions → Demo Timeline Time Event 0:00 Run 2-run-demo.ps1 0:30 Swap staging → production (bad code deployed) 1:00 Production now slow (~1500ms vs ~50ms baseline) ~5:00 Slot Swap Alert fires ~5:04 Agent executes slot swap (rollback) ~5:30 Production restored to healthy state ~6:00 Agent posts to Teams, creates GitHub issue How the Performance Toggle Works The app has a compile-time toggle in ProductsController.cs: private const bool EnableSlowEndpoints = false; // false = fast, true = slow The setup script creates two versions: Production: EnableSlowEndpoints = false → ~50ms responses Staging: EnableSlowEndpoints = true → ~1500ms responses (artificial delay) Get Started 🔗 Full source code and instructions: github.com/microsoft/sre-agent/samples/proactive-reliability 🔗 Azure SRE Agent documentation: https://learn.microsoft.com/en-us/azure/sre-agent/ Technology Stack Framework: ASP.NET Core 9.0 Infrastructure: Azure Bicep Monitoring: Application Insights + Log Analytics Automation: Azure SRE Agent Scripts: PowerShell 7.0+ Tags: Azure, SRE Agent, DevOps, Reliability, .NET, App Service, Application Insights, Autonomous RemediationStop Running Runbooks at 3 am: Let Azure SRE Agent Do Your On-Call Grunt Work
Your pager goes off. It's 2:47am. Production is throwing 500 errors. You know the drill - SSH into this, query that, check these metrics, correlate those logs. Twenty minutes later, you're still piecing together what went wrong. Sound familiar? The On-Call Reality Nobody Talks About Every SRE, DevOps engineer, and developer who's carried a pager knows this pain. When incidents hit, you're not solving problems - you're executing runbooks. Copy-paste this query. Check that dashboard. Run these az commands. Connect the dots between five different tools. It's tedious. It's error-prone at 3am. And honestly? It's work that doesn't require human creativity but requires human time. What if an AI agent could do this for you? Enter Azure SRE Agent + Runbook Automation Here's what I built: I gave SRE Agent a simple markdown runbook containing the same diagnostic steps I'd run manually during an incident. The agent executes those steps, collects evidence, and sends me an email with everything I need to take action. No more bouncing between terminals. No more forgetting a step because it's 3am and your brain is foggy. What My Runbook Contains Just the basics any on-call would run: az monitor metrics – CPU, memory, request rates Log Analytics queries – Error patterns, exception details, dependency failures App Insights data – Failed requests, stack traces, correlation IDs az containerapp logs – Revision logs, app configuration That's it. Plain markdown with KQL queries and CLI commands. Nothing fancy. What the Agent Does Reads the runbook from its knowledge base Executes each diagnostic step Collects results and evidence Sends me an email with analysis and findings I wake up to an email that says: "CPU spiked to 92% at 2:45am, triggering connection pool exhaustion. Top exception: SqlException (1,832 occurrences). Errors correlate with traffic spike. Recommend scaling to 5 replicas." All the evidence. All the queries used. All the timestamps. Ready for me to act. How to Set This Up (6 Steps) Here's how you can build this yourself: Step 1: Create SRE Agent Create a new SRE Agent in the Azure portal. No Azure resource groups to configure. If your apps run on Azure, the agent pulls context from the incident itself. If your apps run elsewhere, you don't need Azure resource configuration at all. Step 2: Grant Reader Permission (Optional) If your runbooks execute against Azure resources, assign Reader role to the SRE Agent's managed identity on your subscription. This allows the agent to run az commands and query metrics. Skip this if your runbooks target non-Azure apps. Step 3: Add Your Runbook to SRE Agent's Knowledge base You already have runbooks, they're in your wiki, Confluence, or team docs. Just add them as .md files to the agent's knowledge base. To learn about other ways to link your runbooks to the agent, read this Step 4: Connect Outlook Connect the agent to your Outlook so it can send you the analysis email with findings. Step 5: Create a Subagent Create a subagent with simple instructions like: "You are an expert in triaging and diagnosing incidents. When triggered, search the knowledge base for the relevant runbook, execute the diagnostic steps, collect evidence, and send an email summary with your findings." Assign the tools the agent needs: RunAzCliReadCommands – for az monitor, az containerapp commands QueryLogAnalyticsByWorkspaceId – for KQL queries against Log Analytics QueryAppInsightsByResourceId – for App Insights data SearchMemory – to find the right runbook SendOutlookEmail – to deliver the analysis Step 6: Set Up Incident Trigger Connect your incident management tool - PagerDuty, ServiceNow, or Azure Monitor alerts and setup the incident trigger to the subagent. When an incident fires, the agent kicks off automatically. That's it. Your agentic workflow now looks like this: This Works for Any App, Not Just Azure Here's the thing: SRE Agent is platform agnostic. It's executing your runbooks, whatever they contain. On-prem databases? Add your diagnostic SQL. Custom monitoring stack? Add those API calls. The agent doesn't care where your app runs. It cares about following your runbook and getting you answers. Why This Matters Lower MTTR. By the time you're awake and coherent, the analysis is done. Consistent execution. No missed steps. No "I forgot to check the dependencies" at 4am. Evidence for postmortems. Every query, every result, timestamped and documented. Focus on what matters. Your brain should be deciding what to do not gathering data. The Bottom Line On-call runbook execution is the most common, most tedious, and most automatable part of incident response. It's grunt work that pulls engineers away from the creative problem-solving they were hired for. SRE Agent offloads that work from your plate. You write the runbook once, and the agent executes it every time, faster and more consistently than any human at 3am. Stop running runbooks. Start reviewing results. Try it yourself: Create a markdown runbook with your diagnostic queries and commands, add it to your SRE Agent's knowledge base, and let the agent handle your next incident. Your 3am self will thank you.Never Explain Context Twice: Introducing Azure SRE Agent memory
In our recent blog post, we highlighted how Azure SRE Agent has evolved into an extensible AI-powered operations platform. One of the most requested capabilities from customers has been the ability for agents to retain knowledge across sessions-learning from past incidents, remembering team preferences, and continuously improving troubleshooting accuracy. Today, we're excited to dive deeper into the Azure SRE Agent memory, a powerful feature that transforms how your operations teams work with AI. Why Memory Matters for AI Operations Every seasoned SRE knows that institutional knowledge is invaluable. The most effective on-call engineers aren't just technically skilled, they remember the quirks of specific services, recall solutions from past incidents, and know the team's preferred diagnostic approaches. Until now, AI assistants started every conversation from scratch, forcing teams to repeatedly explain context that experienced engineers would simply know. The SRE Agent Memory changes this paradigm. It enables agents to: Remember team facts, preferences, and context across all conversations Retrieve relevant runbooks and documentation during troubleshooting Learn from past sessions to improve future responses Share knowledge across your entire team automatically Context Engineering: The Key to Better AI Outcomes At the heart of the memory is a concept we call context engineering, the practice of purposefully curating and optimizing the information you provide to the agent to get better results. Rather than hoping the AI figures things out, you systematically build a knowledge foundation that makes every interaction smarter. The workflow is simple: Identify gaps: Use Session Insights to see where the agent struggled or lacked knowledge Add targeted context: Upload runbooks to the Knowledge Base or save facts with User Memories Track improvement: Review subsequent sessions to measure whether your additions improved outcomes Iterate: Continuously refine your context based on real session data This feedback loop transforms ad-hoc troubleshooting into a systematically improving process, where each session makes future sessions more effective. Memory Components at a Glance The memory consists of three complementary components that work together to give your agents comprehensive knowledge: 🧠 User Memories: Quick Chat Commands for Team Knowledge Save facts, preferences, and context using simple chat commands. User Memories are ideal for team standards, service configurations, and workflow patterns that should persist across all conversations. Key benefits: ✅ Instant setup-no configuration required ✅ Managed directly in chat with #remember, #forget, and #retrieve commands ✅ Shared across all team members automatically ✅ Works across all conversations and agents Example commands: #remember Team owns app-service-prod in East US region #remember For latency issues, check Redis cache first #remember Production deployments happen Tuesdays at 2 PM PST When you save a memory, it's instantly available across all your team's conversations. The agent automatically retrieves relevant memories during reasoning, no additional configuration needed. Saving team knowledge with the #remember command Use #retrieve to search and display your saved memories: Retrieving saved memories with the #retrieve command 📚 Knowledge Base: Direct Document Uploads for Runbooks and Guides Upload markdown and text files directly to the agent's knowledge base. Documents are automatically indexed using semantic search and available for agent retrieval during troubleshooting. The Knowledge Base uses intelligent indexing that combines keyword matching with semantic similarity. Documents are automatically split into optimal chunks, so agents retrieve the most relevant sections, not entire documents. Key benefits: ✅ Supports .md and .txt files (up to 16MB per file) ✅ Automatic chunking and semantic indexing ✅ Simple file upload interface ✅ Instant availability after upload Best for: Static runbooks, troubleshooting guides, internal documentation, and configuration templates. Navigate to Settings > Knowledge Base to access document management. There you will find Add File, allows you to upload txt and md file(s) and Delete, allows you to delete individual or bulk files. 📊 Session Insights: Automated Analysis of Your Troubleshooting Sessions Get automated feedback on your troubleshooting sessions with timelines, performance analysis, and key learnings. Session Insights help you understand what happened, learn from mistakes, and continuously improve. Key benefits: ✅ Automatic analysis after conversations complete ✅ Chronological timeline of actions taken ✅ Performance scoring with specific improvement suggestions ✅ Key learnings for future sessions Navigate to Settings > Session Insights to view your troubleshooting analysis: Session Insights dashboard showing analysis of past troubleshooting sessions You can also manually trigger insight generation for any conversation by clicking the Generate Session Insights icon in the chat footer: Manually triggering Session Insights generation Each insight includes: Timeline: A chronological narrative showing what actions were taken and their outcomes What Went Well: Highlights correct understanding and effective actions Areas for Improvement: Shows what could be done better with specific remediation steps Key Learnings: Actionable takeaways for future sessions Investigation Quality Score: Sessions rated on a 1-5 scale for completeness How Azure SRE Agent Use Memory: The SearchMemory Tool During conversations, incident handling, and scheduled tasks, Azure SRE Agents search across memory sources to retrieve relevant context using the SearchMemory tool. Enabling Memory Retrieval in Custom Sub-Agents When building custom sub-agents with the Sub-Agent Builder, you can enable memory retrieval by adding the SearchMemory tool to your sub-agent's toolset. This allows your custom automation to leverage all the knowledge stored in User Memories and the Knowledge Base. How it works: In the Sub-Agent Builder, add the SearchMemory tool to your sub-agent's available tools The tool automatically searches across all memory sources using intelligent retrieval Your sub-agent receives relevant context to inform its responses and actions This means your custom sub-agents, whether handling specific incident types, automating runbook execution, or performing scheduled health checks, can all benefit from your team's accumulated knowledge. Choosing the Right Memory Type Feature User Memories Knowledge Base Setup Instant (chat commands) Quick (file upload) Management Chat commands Portal UI Content Size Short facts Documents (up to 16MB) Best Use Case Team preferences Static runbooks Team Sharing ✅ Shared ✅ Shared Quick guidance: User Memories: Short, focused facts (1-2 sentences) for immediate team context Knowledge Base: Well-structured documents with clear headers for procedural knowledge Getting Started in Minutes 1. Start with User Memories Open any chat with your Azure SRE Agent and save immediate team knowledge: #remember Team owns services: app-service-prod, redis-cache-prod, and sql-db-prod #remember For latency issues, check Redis cache health first #remember Team uses East US for production workloads That's it, these facts are now available across all conversations. 2. Upload Key Documents Add critical runbooks and guides to the Knowledge Base: Navigate to Settings > Knowledge Base Upload .md or .txt files Files are automatically indexed and available immediately 3. Review Session Insights After troubleshooting sessions, check Settings > Session Insights to see what went well and where the agent needs more context. Use this feedback to identify gaps and add targeted memories or documentation. Best Practices for Building Agent Memory Content Organization Keep memories focused and specific Use consistent terminology across your team Avoid duplication, choose one source of truth for each piece of information Security Never store: ❌ Credentials, API keys, or secrets ❌ Personal identifiable information (PII) ❌ Customer data or logs ❌ Confidential business information Maintenance Regularly review and update memories Remove outdated information using #forget Consolidate duplicate entries Use #retrieve to audit what's been saved The Impact: Smarter Troubleshooting, Lower MTTR The Azure SRE Agent memory delivers measurable improvements: Faster troubleshooting: Agents immediately understand your environment and preferences Reduced toil: No more repeatedly explaining the same context Institutional knowledge capture: Critical team knowledge persists even as team members change Continuous improvement: Each session makes future sessions more effective By systematically building your agent's knowledge foundation, you create an operations assistant that truly understands your environment, reducing mean time to resolution (MTTR) and freeing your team to focus on high-value work. Ready to Get Started? Azure SRE Agent home page Product documentation Pricing information Demo recordings What's Next? We're continually enhancing the memory based on customer feedback. Your input is critical, use the thumbs up/down feedback in the agent, or share your thoughts in our GitHub repo. What operational knowledge would you like your AI agent to remember? Let us know! This blog post is part of our ongoing series on Azure SRE Agent capabilities. See our previous post on automation, integration, and extensibility features.
