Why Did Our Azure Bill Change? Investigating Cloud Spend with Azure SRE Agent
The question after the cost chart The monthly cost review usually starts with a straightforward observation: Azure spending changed. Then come the questions that take the time. Which subscription accounts for the increase? Is it an existing workload getting more expensive, or something that only started generating charges this month? Did a reduction somewhere else hide part of the increase? And which optimization recommendations are worth taking to a workload owner? Azure already has the underlying information. Cost Management provides spending data. Azure Advisor provides optimization recommendations. Resource inventory provides context about what those resources are. The work is connecting those sources into an explanation that someone can check and act on. Conversational Cost Analytics Agent: a solution built around Azure SRE Agent that lets operations and FinOps leads start with a question, investigate cost drivers, identify optimization opportunities, and generate reusable reports. The solution can also generate proactive alerts and scheduled reports. Scope of this post: The repository contains a prototype, an importable dashboard template, reusable skills, and example report screenshots. These demonstrate the reporting approach; they are not evidence of measured customer savings, verified production readiness, or a completed deployment in your environment. The prompts below are examples to try, not a transcript of a measured production investigation. What we built This is a custom cost-investigation workflow on Azure SRE Agent, not a separate billing system or a claim that every feature is available out of the box. The solution brings together three pieces: An analytical skill that describes how to compare spending, calculate resource contributions, identify newly billed resources, and handle incomplete periods. A query-reference skill that records the Cost Management query patterns, response shapes, aggregation rules, and SRE Agent tool mappings used by the dashboard workflow. An HTML Live Report template that presents subscription selection, summary metrics, six-month trends, and expandable resource-level comparisons. The distinction matters. A template determines how findings appear. A skill determines how the agent should investigate. Neither replaces access to the underlying data. The agent still needs scoped permissions, supported tools, and a successful data refresh before the report can represent your subscriptions.' How the investigation works The workflow starts by discovering the subscriptions the agent can access. It then queries Cost Management for historical totals and current- and previous-month resource-level details, aggregates the results, and populates the report template. Source or component Role in the investigation Azure Cost Management Billing-based cost data and historical comparisons Azure Resource Graph Resource inventory and metadata enrichment, not cost data Azure Advisor Optimization recommendations to review alongside spending Reusable agent skills Investigation rules, query patterns, and interpretation guardrails Live Report template Interactive presentation of the refreshed data snapshot How cost data is collected: The agent sends a request to Azure Cost Management to retrieve spending data. The request uses HTTP POST, so the kit runs it through RunAzCliWriteCommands, even though it only reads data. Reading costs does not change resources: This query does not resize, delete, or modify resources. Still, configure permissions and tools carefully to limit what the agent can do. The report shows the most recently refreshed data: Opening the report displays the data collected during its most recent refresh. It does not automatically request updated billing data. Fetching data directly from the report requires a connector: A supported connector gives the report access to a cost-query tool. The report cannot directly use the agent’s built-in CLI tools. You can start without a connector: Use the report with periodically refreshed data, and add a connector later if you need to fetch updates directly from the report. Four questions to explore in the report 1. Where is spending heading? Start with the broad view: The overview brings subscription selection, summary KPIs, previous-month actuals, and an adjustable cost trend into one view. Select the number of months you want to compare, up to 12, and expand any month to explore the services behind the spending changes. Figure 1. Historical prototype snapshot, not current pricing or a verified financial result. The current-month KPI is a projection; the screenshot does not establish that the partial-month trend bar uses the same basis. Reconcile those values and use consistent labels before relying on the comparison. This is also where an apparently simple comparison can become misleading. Ten days of current-month spend and a full previous month are not comparable periods. For an in-progress month, ask for matched month-to-date periods when investigating change. If the report displays a full-month run-rate projection, keep it labeled as a projection. Do not describe it as a Cost Management forecast unless that forecast was actually retrieved. The useful outcome is not just a larger or smaller bar. It is an explicit statement of the period, cost basis, currency, and freshness behind that bar. 2. Which resources explain the change? Once the subscription-level movement is clear, move to the contributors: "Rank the resources driving increases and decreases. Separate newly billed resources from existing resources whose costs changed." The resource view compares spending at the resource level, highlights resources appearing in the current period’s billing data, and supports expandable service breakdowns. Figure 2. Historical prototype comparison. The screenshot's "New Resources" label means newly observed in billing, not confirmed creation. In this example workflow, current-month "Full" values are projections, not completed-month actuals. The agent can help break down cost changes that might otherwise require multiple spreadsheets and time-consuming manual comparisons: New charges do not always mean new resources: An existing resource may start generating charges after a period of inactivity. Costs may also rise because of autoscaling, longer VM running hours, increased storage usage, more read/write operations, or billable data transfers. These changes can be difficult to spot in summary reports. Newly created resources can add to spending: The agent can highlight resources appearing in the current period’s billing data. Resource inventory or deployment history is needed to confirm whether they were newly created or simply started generating charges. One resource can have several cost entries: A resource may appear in multiple billing records for different services or meters. Adding the relevant entries together helps show its total cost and contribution to the change. The net change can hide larger movements: For example, if one resource’s cost increases by $800 while another decreases by $300 over comparable periods, the net increase is $500. Showing both changes reveals the increase and the offsetting reduction. These figures are illustrative. Finding the cost driver is not the same as finding the cause: Identifying which resource contributed to an increase is the first step. Explaining why it happened may require additional evidence about usage, configuration changes, pricing, or other factors. 3. Which optimization opportunities deserve attention? The next question is usually practical: "Show Azure Advisor cost optimization recommendations for the subscriptions in scope. Separate estimated savings from confirmed spending reductions." The repository includes an example Advisor optimization view for reviewing rightsizing and commitment-related opportunities alongside the cost investigation. Figure 3. Historical prototype optimization view. Displayed savings are estimates, not realized savings; low CPU alone does not establish that shutdown is appropriate. A displayed zero for reservations requires a successful, authorized query before it can be interpreted as no reservations. The useful next step is to review the findings with the workload owner, not to make an automatic purchase or resize a resource. Rightsizing can change the baseline for a reservation or savings plan decision. Recommendations can also overlap, so adding every displayed savings figure may overstate the opportunity. Billing and commitment visibility depend on the identity's actual access; subscription resource access does not automatically expose every reservation or savings plan dataset. An unavailable dataset should be reported as unavailable. It should never silently become "no recommendations" or "no commitments." 4. What might we be paying for without using? Finally, investigate cleanup candidates: "Identify potentially orphaned resources and show the evidence that makes each one a candidate for owner review." The example orphaned-resource view covers candidates such as unattached disks, stale snapshots, unassociated public IP addresses and network interfaces, and empty App Service plans. Figure 4. Example cleanup-candidate inventory. An apparent orphan is an investigation finding, not deletion approval. An unattached disk may be retained for recovery. A snapshot may be subject to a retention requirement. A network resource may be reserved for a planned deployment. The report should help the owner answer those questions by identifying the resource, its state, and any observed charges. It should not turn "not currently attached" into "safe to delete." The intended configuration is read-only reporting and investigation. Validate the identity's effective permissions and enabled tools before use; the template alone does not enforce that boundary. Any cleanup requires separate validation and explicit approval. Have a meaningful cost conversation with the agent A cost dashboard remains useful. It shows the shape of spending and gives people a shared view. However, a conversational approach takes this further by helping users investigate what is driving spending, identify key contributors, distinguish new charges from changes in existing resources, and explore cost optimization opportunities. Start with a simple cost question, then use follow-up prompts to explore what changed, which resources contributed, and what needs further investigation. Example prompts include: “Which subscriptions and resources explain the largest cost increases compared with last month?” “Why did costs increase this month?” “Which resources contributed most to the increase?” “Is this newly billed spending or higher costs from existing resources?” “Compare the last three months and show the biggest changes.” “Create a report focused on this subscription and its top cost drivers.” “Show Azure Advisor savings opportunities and orphaned resources that require owner review.” “Refresh this report every Monday morning.” With the required data access and tools configured, the agent can help rerun analyses, narrow the scope, and generate reports focused on specific cost questions. Scheduled reports and cost alerts require supported capabilities, appropriate permissions, and separate configuration. Check findings against source data before making decisions or changing resources. Try it in your environment Start with one subscription whose costs you already understand. That makes it easier to reconcile the first report before expanding the scope. Review the kit. Start with the repository overview Add the three skills from the skills folder, and upload the cost analysis playbook as a knowledge source. Configure scoped access. The kit calls for Cost Management Reader and Reader on each target subscription for cost queries and inventory enrichment. Validate any additional billing or commitment access separately. Bootstrap the report. Submit the first-run prompt from the bootstrap prompts and provide the HTML report template. Validate the result. Reconcile a completed month's totals against Cost Management using the same scope, currency, and cost basis. Check resource aggregation, labels, and missing-data states. Configure refresh only after validation. The kit includes a suggested Monday 07:00 UTC refresh prompt. Confirm scheduling support, permissions, and failure handling in your environment before relying on it. Using this workflow may incur Azure SRE Agent charges and other applicable service charges. Review Azure SRE Agent pricing and billing and the pricing for the services you configure before running a pilot. This post does not describe a free service offering. Further reading Azure SRE Agent overview Azure Cost Management overview Azure Cost Management Query API Azure Advisor cost recommendations Azure Resource Graph overviewAnnouncing flex nodes for AKS (preview) in Azure portal
Explore how AKS flex nodes can address regional CPU and GPU capacity constraints, bring distributed compute under one control plane, and support on-premises workload placement for data residency, now with the Azure portal experience to simplify onboarding.Announcing the Public Preview of the New App Service Quota Self-Service Experience
Update 10/30/2025: The App Service Quota Self-Service experience is back online after a short period where we were incorporating your feedback and making needed updates. As this is public preview, availability and features are subject to change as we receive and incorporate feedback. What’s New? The updated experience introduces a dedicated App Service Quota blade in the Azure portal, offering a streamlined and intuitive interface to: View current usage and limits across the various SKUs Set custom quotas tailored to your App Service plan needs This new experience empowers developers and IT admins to proactively manage resources, avoid service disruptions, and optimize performance. Quick Reference - Start here! Leverage the new self-service experience to increase your quota automatically. If your deployment requires quota for ten or more subscriptions, then file a support ticket with problem type Quota following the instructions at the bottom of this post. If any subscription included in your request requires zone redundancy (note that most Isolated v2 deployments require ZR), then file a support ticket with problem type Quota following the instructions at the bottom of this post. Self-service Quota Requests For non-zone-redundant needs, quota alone is sufficient to enable App Service deployment or scale-out. Follow the provided steps to place your request. 1. Navigate to the Quotas resource provider in the Azure portal 2. Select App Service (Public Preview) Navigating the primary interface: Each App Service VM size is represented as a separate SKU. If the intention is to be able to scale up or down within a specific offering (e.g., Premium v3), then equivalent number of VMs need to be requested for each applicable size of that offering (e.g., request 5 instances for both P1v3 and P3v3). As with other quotas, you can filter by region, subscription, provider, or usage. Note that your portal will now show "App Service (Public Preview)" for the Provider name. You can also group the results by usage, quota (App Service VM type), or location (region). Current usage is represented as App Service VMs. This allows you to quickly identify which SKUs are nearing their quota limits. Adjustments can be made inline: no need to visit another page. This is covered in detail in the next section. Total Regional VMs: There is a SKU in each region called Total Regional VMs. This SKU summarizes your usage and available quota across all individual SKUs in that region. There are three key points about using Total Regional VMs. You should never request Total Regional VMs quota directly - it will automatically increase in response to your request for individual SKU quota. If you are unable to deploy a given SKU, then you must request more quota for that SKU to unblock deployment. For your deployment to succeed, you must have sufficient quota in the individual SKU as well as Total Regional VMs. If either usage is at its respective limit, then you will be unable to deploy and must request more of that individual SKU's quota to proceed. In some regions, Total Regional VMs appears as "0 of 0" usage and limit and no individual SKU quotas are shown. This is an indication that you should not interact with the portal to resolve any quota-related issues in this region. Instead, you should try the deployment and observe any error messages that arise. If any error messages indicate more quota is needed, then this must be requested by filing a support ticket with problem type Quota following the instructions at the bottom of this post so that App Service can identify and fix any potential quota issues. In most cases, this will not be necessary, and your deployment will work without requesting quota wherever "0 of 0" is shown for Total Regional VMs and no individual SKU quotas are visible. See the example below: 3. Request quota adjustments Clicking the pen icon opens a flyout window to capture the quota request: The quota type (App Service SKU) is already populated, along with current usage. Note that your request is not incremental: you must specify the new limit that you wish to see reflected in the portal. For example, to request two additional instances of P1v2 VMs, you would file the request like this: Click submit to send the request for automatic processing. How quota approvals work: Immediately upon submitting a quota request, you will see a processing dialog like the one shown: If the quota request can be automatically fulfilled, then no support request is needed. You should receive this confirmation within a few minutes of submission: If the request cannot be automatically fulfilled, then you will be given the option to file a support request with the same information. In the example below, the requested new limit exceeds what can be automatically granted for the region: 4. If applicable, create support ticket If automatic quota fulfillment fails, and it recommend you Create a support request, then follow the steps given in the at the end of this post. Known issues The self-service quota request experience for App Service is in public preview. Here are some caveats worth mentioning while the team finalizes the release for general availability: Closing the quota request flyout window will stop meaningful notifications for that request. You can still view the outcome of your quota requests by checking actual quota, but if you want to rely on notifications for alerts, then we recommend leaving the quota request window open for the few minutes that it is processing. Some SKUs are not yet represented in the quota dashboard. These will be added later in the public preview. The Activity Log does not currently provide a meaningful summary of previous quota requests and their outcomes. This will also be addressed during the public preview. As noted in the walkthrough, the new experience does not enable zone-redundant deployments. Quota is an inherently regional construct, and zone-redundant enablement requires a separate step that can only be taken in response to a support ticket being filed. Quota API documentation is being drafted to enable bulk non-zone redundant quota requests without requiring you to file a support ticket. Create a support request Important: Most App Service quota requests can be completed through the self-service experience without creating a support request. If you're prompted to create one, select Severity B or Severity C. Requests submitted at these severity levels will enter the automated quota-processing workflow. Severity A requests are not currently eligible for this automation, and selecting a higher severity does not typically expedite a quota increase request. While we are continuously improving the system to automatically process quota requests, there are certain scenarios you might need to create a support request: Automatic fulfillment request failed on quota blade. Deployment requires zone-redundancy You want to make bulk request for ten or more subscriptions When creating a support request, select your Issue type as “Service and subscription limits (quotas)” and Quota type as “Function or Web App (Windows and Linux)”. Select Next. You can then fill in your quota requirements by clicking on “Enter details”. There are 4 mandatory fields you must provide in your request: Region – Quota limits are based on region. If you are facing quota limits in one region, you can always try deployment in a geographically paired region. For example, West US 2 and West Central US are paired regions. East Asia (Hong Kong) and Southeast Asia (Singapore) are also paired regions. See Azure cross-region replication pairings for all geographies for more information. Deployment type – This is another important criterion when submitting quota request. If you are not sure which deployment type your App Service Plan is using, you can check it here on the portal: App >> App Service Plan >> Zone redundant >> Enabled\Disabled. App Service plan – Each SKU in your subscription and the region selected above, will have its own limit. Choose the SKU based on your deployment requirement. You will be able to see the current usage and the limit on that SKU. New limit – You must submit the new limit that you want for the SKU selected above. Do not add the increment value. The new limit must be higher than the existing limit. If you choose to create a support ticket, then you will interact with the capacity management team for that region. This is a 24x7 service, so requests may be created at any time. Once you have filed the support request, you can track its status via the Help + support dashboard. Note for Logic Apps You can now self-serve your Logic App quota requirements using the same App Services quotas blade. You must choose one of the Logic App SKUs (WS1, WS2, WS3) when making the request, and it will be processed in the same way App Services requests are processed. We want your feedback! If you notice any aspect of the experience that does not work as expected, or you have feedback on how to make it better, please use the comments below to share your thoughts!Plan Your Java Upgrade Ahead of Microsoft’s 2027 Support Milestones
In April 2021, we announced the Microsoft Build of OpenJDK. We were starting with a preview of our Java 11 builds for Windows, Linux, and macOS. Today, if you’re planning a Java upgrade on Azure, you have several generations of that work to build on: 11, 17, 21, and now 25. Java 8 has been part of that work too, with help from the fine folks at the Eclipse Foundation’s Eclipse Adoptium project. Microsoft has supported Java 8 on Azure using their Eclipse Temurin builds of OpenJDK 8, rather than producing a Microsoft Build of OpenJDK 8. That collaboration goes back to the launch: our first Java 11 builds also used Adoptium’s build scripts and quality assurance suite. Supporting those releases has meant years of updates. Now it’s time to plan the next move. The Microsoft Build of OpenJDK support roadmap lists September 2027 as the earliest end-of-support date for Java 11 and 17. That’s a distribution-wide milestone: if you use those Microsoft builds on a VM, in a container, on Kubernetes, on Azure platform services, or evem outside Azure, it belongs in your upgrade planning. These dates are initial targets, and Microsoft may extend them. However, consider taking action now. Azure services also have their own deadlines. Azure App Service and Azure Functions support for Java 8, 11, and 17 ends on September 1, 2027. Existing applications will continue to run, but the affected Java versions will no longer receive security updates, improvements, or related customer support. If that’s your application, the practical question is how to get to Java 21 or 25 while keeping it working. GitHub Copilot modernization gives you a tool: the Modernize CLI, which can assess your code, help plan the upgrade, and actually perform and apply changes for you to review. Continue reading on the Java at Microsoft blog...Introducing a Guided Copilot Experience for Building Azure Apps in VS Code
Today we're previewing a new way to build cloud apps with GitHub Copilot in VS Code: a guided Copilot experience that takes you from idea to a deployed Azure app through a structured, predictable workflow instead of a free-form chat session that may or may not land where you need it to. The problem: Copilot is powerful, but unpredictable Ask Copilot to "build me a Node.js API on Azure Functions with a Postgres database" today, and you might get a working project, or you might not. Sometimes Copilot scaffolds the app but skips infrastructure. Sometimes it generates a deployment script that fails halfway through. Sometimes it forgets what you were building three prompts later. The underlying model is capable. The problem is the shape of the experience: an open-ended chat with no checkpoints, no guardrails, and no consistent path from "I have an idea" to "it's running in Azure." How it works The guided Copilot experience adds to that open-ended flow with three clear, explicit stages: 1. Project scaffolding: Describe your app in plain language. Copilot proposes an architecture and asks for anything that is missing (e.g., language, app type, Azure services) through simple forms and pickers, not more back-and-forth chat. You review and approve the plan before any code is written. 2. Local development: Your project is scaffolded then Copilot checks for the runtimes, emulators, and tools you'll need and helps you install what's missing, so the project runs locally from the very first launch. Debug configurations are wired up automatically, so you can test and iterate on your app before it ever touches Azure. 3. Deployment: Copilot shows the tools that will be used for deployment along with a summary of all resources that will be created and a cost estimate so you can be confident in what you're getting. Infrastructure files are created for your app to ensure a consistent and reproducible deployment environment for testing and staging before going to production. Deployment runs through the same reliable tooling used across Azure (az and azd), so if something goes wrong, you get a clear explanation and a concrete next step instead of a wall of CLI output. Once your app is live, the guided experience doesn't disappear, Copilot retains context about your project's architecture, so coming back later to add a service or make a change picks up right where you left off. 🎉 What this means for how you work Determinism over guesswork. The same input reliably produces the same kind of outcome, so no more wondering which "mode" Copilot is going to be in. First try success. Projects are built to run and deploy on the first attempt, not after several rounds of manual repair. Structure over chat walls. Decisions that matter such as architecture choices, missing configuration, and deployment targets are surfaced through real UI, so you're not parsing paragraphs of text to figure out what Copilot needs from you. Stay in VS Code. The entire journey, from planning through deployment, happens without leaving your editor. What's in the initial preview The first version focuses on JavaScript and TypeScript projects, including web apps, Azure Functions, Container Apps, and Static Web Apps, with support for PostgreSQL, Azure Storage, Azure Key Vault, and Azure OpenAI. .NET and Python support are on the roadmap for a future release. Nothing about your existing Copilot workflows changes, the guided experience is an additional, opinionated path alongside the free form chat you already know, for when you want a reliable, structured way to go from zero to deployed. What's next This is an early look at a broader shift in how we think about AI-assisted development on Azure: less "ask and hope," more structured collaboration with clear checkpoints and real UI where it counts. We're continuing to expand language support, refine the deployment experience, and incorporate feedback from developers using it in the wild. Install the Azure Tools extension pack and open an empty folder to try the guided Copilot experience and let us know what you think. File issues or share feedback on the GitHub repo.
Enterprise Adoption Guide for Azure SRE Agent, Part 1: Deploying Your First Azure SRE Agent
We have learned a lot from talking with teams that are bringing Azure SRE Agent into their day-to-day operations. Across those conversations, we have seen a common set of questions emerge as teams move from exploring an initial use case to adopting SRE Agent more broadly. Some teams are starting with a simple question: “Where should we begin?” Others have already seen value in reducing operational toil and improving incident response and are asking what comes next. How do we expand from one successful use case to more workflows? Can multiple teams share the same agent safely? When should we use one SRE Agent versus multiple agents? How should agents collaborate? And how do we deploy and govern them consistently across the enterprise? This two-part guide shares the patterns we have seen and our recommendations for navigating that journey, from the first use case to broader enterprise adoption. Our recommendation is to start with one operational problem your team understands, learn what the agent needs to solve it effectively, and expand from there. The goal is not to design your enterprise topology on day one. It is to make each next decision with better evidence. Part 1: Deploying Your First Azure SRE Agent Which operational problem should we start with? How do I run the first use case? Should I use SRE Agent’s built-in capabilities or customize it? How do we turn a successful first use case into more automated operational workflows? Part 2: From One Team to Enterprise-Wide Adoption How do we share an SRE Agent with more users? How do we expand from one team to multiple teams, and do we need one SRE Agent or multiple? How do we codify, deploy, and govern SRE Agents at enterprise scale? Part 1 focuses on proving and expanding the operational value. Part 2 addresses the questions that appear when more users, teams, and agents become involved. 1. Which operational problem should we start with? A strong first use case is a recurring, bounded problem that the team feels today and understands well enough to judge. Focus on work that affects reliability, consumes engineering time, slows incident response, or creates avoidable cost. Start with the operational pain your team feels most. Choose one well-bounded problem that matters to the team and is familiar enough to judge. If you are unsure where to begin, start by asking SRE Agent to investigate an issue or incident your team is already looking at. Give it the observed symptom or question, let it bring together telemetry, dependencies, source code, and recent changes, and see whether it can identify where the problem is likely occurring. That investigation can include root-cause analysis, deployment correlation, Azure resource health, and dependency analysis. The examples below are some of the use cases we have seen customers implement. Alert investigation: correlate alerts with telemetry, dependencies, source code, and recent changes to explain what happened and identify the next step. Incident auto-triage: collect the initial evidence, identify likely ownership and severity, remove known false positives, and prepare the incident for the responding team. Proactive availability: perform proactive health checks, reliability risks, degraded signals, or unhealthy dependencies before they become customer-facing incidents. Deployment correlation: correlate a failure or degradation with recent deployments, configuration changes, feature releases, and infrastructure updates. Production deployment validation: evaluate post-deployment telemetry, logs, dependencies, and service-health criteria to confirm that a production release is healthy. FinOps proactive cost optimization: identify unexpected cost changes, underused resources, and rightsizing opportunities before spend becomes a larger problem. Certificate expiry monitoring: identify certificates approaching expiration, determine the affected services and owners, and initiate the appropriate follow-up. 2. How do I run the first use case? Should I use SRE Agent’s built-in capabilities or customize it? Once the team has selected the operational pain it wants to address, the next step is to prove that SRE Agent can improve the outcome. Two questions matter at this stage: how should we set up SRE Agent to execute the first use case, and do we need to customize beyond its built-in capabilities? Keep the first use case deliberately small. Connect only the context required to solve the selected problem. For deployment validation, that may be the relevant telemetry source, code repository, deployment history, and runbook, not every repository, monitoring system, or resource the team may eventually use. For more guidance on setting up the first SRE Agent, see Getting the best out of Azure SRE Agent. Begin with diagnostic or read-only access. Use the agent’s managed identity and Azure RBAC as the starting point for Azure resource access. Grant only the resource scopes and permissions required for the use case. As the workflow demonstrates a need to perform write actions, add the narrowest permission at the narrowest practical scope rather than granting broad access up front. Test the agent against a few recent historical incidents the team understands. Use these cases to confirm that the agent can reach the right telemetry, code, deployment history, and operational knowledge, and to expose missing context before a live incident. Then use SRE Agent alongside the team’s normal response process when a real issue occurs. This is where teams begin to build trust: the agent surfaces the relevant evidence, identifies where the problem is likely occurring, connects it to a dependency or recent change, explains its reasoning, and helps responders reach the diagnosis faster. Define the outcomes you want to accomplish. Measure the outcome in the system that already records it, for example, mean time to resolution in the incident platform, alert volume or false-positive rate in monitoring, deployment failure or rollback rate in CI/CD. Choose the measure that matches the pain that justified the use case. Establish a baseline, then use an SRE Agent live report or an existing dashboard to track the result. 2.1 When do I customize SRE Agent, and do I need a skill, custom subagent, or tool? Always start small with SRE Agent’s built-in capabilities. Then add only what the workflow needs. Start with the expert skills and tools that come with SRE Agent. Out of the box, SRE Agent combines Azure-specific product knowledge and built-in operational capabilities for diagnosing issues and finding probable root causes across Azure. This expertise spans all of Azure, including compute, storage, networking, data, monitoring, and management services. Its native integrations connect Azure Monitor, Application Insights, and Log Analytics, GitHub, and Azure DevOps so the agent can correlate infrastructure state, telemetry, deployments, work items, and source code. General-purpose tools, including Azure CLI, Kusto queries, and code analysis, let it gather deeper evidence within its assigned permissions. For example, it can trace network connectivity, troubleshoot gateway and load-balancing paths, query application telemetry and logs, and correlate a degradation with a recent code or deployment change. Add a custom skill when SRE Agent needs to understand how your environment or organization works. Before writing a new skill, browse the plugin marketplace and the public Azure SRE Agent Plugins repository to see whether an existing plugin covers the scenario. A skill can teach the agent how to use and interpret a third-party telemetry source such as Dynatrace, which dashboards or signals matter, how your runbooks and escalation processes work, or how your organization performs a workflow such as pull-request validation, release-risk analysis, or compliance review. Even with a native integration, adding your operating instructions as a skill can help the agent find the right evidence faster. Add a custom tool when part of the workflow must run consistently. Use deterministic code when the agent must perform the same calculation, execute a predefined query, call an external API, or generate a report in an exact format every time. A skill explains when and how your organization performs the task; a tool performs the required action consistently. Prefer an existing built-in or connector tool before adding a custom Python tool, HTTP or Kusto tool, or MCP connector. Use a skill and tool together when the workflow needs both organizational guidance and deterministic execution. For example, a skill can describe how your organization evaluates deployment risk, while a tool calculates the risk score or generates the required report. For most workflows, built-in capabilities plus a custom skill are sufficient. Add a custom tool only when the workflow requires repeatable execution, external-system access, or an exact output. 2.2 When do I need a custom subagent? A custom subagent is a specialist within SRE Agent designed for a focused operational workflow. For most use cases, start with a skill. A skill is enough when the main SRE Agent needs to understand and apply your team’s instructions as part of a broader conversation or investigation. Create a custom subagent when you want to control the scope of that workflow, package a focused set of instructions, skills, and tools, test or invoke it independently, or connect it to a trigger. The use case itself does not determine the choice. The same deployment-validation guidance can remain a skill when an engineer asks the main agent to apply it during an investigation. Package that skill in a deployment-validation agent when the workflow should stay within a defined scope, use selected tools, and run after a deployment through an incident, schedule, HTTP trigger, or another workflow. Start with a skill. Use a custom subagent when the workflow needs tighter scope, focused instructions and tools, independent invocation, or trigger-based automation. 3. How do we turn a successful first use case into more automated operational workflows? Once the first use case proves the value, look at what the team still does manually around it. Where are engineers repeatedly gathering the same evidence, running the same diagnostics, monitoring the same risks, preparing the same updates, or handing off the same follow-up work? Use live reports to bring together incident, service-health, reliability, and cost signals so recurring patterns and remaining operational toil are easier to see. When the team understands the work and the result is repeatable, it is ready to become an automated workflow. Begin with the context already connected to the agent. Reuse the same telemetry, repositories, runbooks, resource access, and team knowledge where they apply. Add another source only when the workflow needs evidence the agent cannot currently reach. There are different ways to start that automation. Use a: Response plan when an incident or alert should start the investigation Scheduled task for recurring health, capacity, compliance, or reporting work HTTP trigger when a pipeline, external monitoring system, or another workflow should invoke the agent Start these workflows in Review mode, where a person can evaluate the agent’s reasoning and proposed actions. Autonomous mode is a good fit for actions the team has already approved its operators to perform without case-by-case review. Start with a human in the loop, build confidence in the workflow, and move approved actions to Autonomous mode when the team is comfortable allowing the agent to perform them on its own. Start automating the work the team now understands. Use the right trigger for each workflow, and use live reports to make the next source of operational toil visible. Continue to Part 2: From One Team to Enterprise-Wide Adoption Every team has operational work it wishes were faster, more consistent, or less manual. What is the recurring operational problem your team would most like to solve with SRE Agent? And what could we build or improve to make it easier to turn that problem into a trusted workflow? In part 1, we explored how your organization can go from zero to one with adoption of Azure SRE Agent to address this operational toil in your organization. We'd love for you to share your experience and ideas in the comments below. Your feedback will help us understand where teams need more guidance, better starting points, or new product capabilities. To continue on to scaling Azure SRE Agent across your enterprise, go on to: Part 2: From One Team to Enterprise-Wide Adoption. Next steps Continue to Part 2: From One Team to Enterprise-Wide Adoption. Open Azure SRE Agent and start your first use case. Create and set up an Azure SRE Agent. Explore the public SRE Agent templates for infrastructure as code, export, verification, and CI/CD examples.Meet the Hosted Skills Canvas: Build, Run, and Debug in GitHub Copilot
Build, run, and debug event-driven AI apps right inside GitHub Copilot. The new Azure Functions Hosted Skills canvas brings instructions, triggers, and live results into one workspace—so you can spend less time switching tools and more time building.
Bring Azure SRE Agent into your developer workflow with the new canvas
You’re on call. A customer-reported incident lands, Azure SRE Agent picks it up from your incident management system, investigates, and gives you a starting diagnosis. If you’re like me, you want to check that diagnosis yourself, and you want to do it where you already work: in GitHub Copilot, with your code, your tools, and your own access to the data. The new Azure SRE Agent canvas puts the investigation right next to your Copilot chat. A canvas is an interactive panel beside the conversation, with context and actions you and Copilot can both use. You get SRE Agent’s findings plus everything SRE doesn’t have: your code, your private data sources, and what you know about the app. In the GitHub Copilot App, sessions are on the left, Copilot chat is in the middle, and the SRE canvas is on the right. You can use the same canvas in Copilot CLI sessions that support canvases. I now have SRE's findings alongside my own context in Copilot chat. I can keep digging here and decide what to send back to SRE, and when. Let me show you the inner loop with an incident from September 22. It’s a Contoso demo, but it’s modeled on a real one we hit in production. Gateway health looked available while orders and payments failed. I'll use SRE's diagnosis as a starting point, check the historical telemetry with my own authorized access, and work out what the evidence does and doesn't establish. I’ll share findings with the team, and anything worth keeping goes back to SRE Agent. Start with the incident With the plugin enabled, I prompt Copilot to open Azure SRE Agent and choose my saved demo1-dt-snow favorite. I can also discover agents in a subscription I choose or connect through a caller-authorized external URL. For this investigation, I open INC0922001, "Trading order and payment requests failing while gateway health remains available," and choose Connect this thread. Selecting a row lets me browse; connecting makes it the investigation for this conversation. The September incident as recorded in SRE Agent. Then I ask: Summarize this connected September 22 investigation. Separate the reported diagnosis from observed evidence. Give the incident time window, and list what still needs independent verification. The SRE record reports a stopped PostgreSQL database. It says the gateway passed its health check, but Orders and payments requests failed after about 15 seconds. That's useful context, but a passing health check doesn't mean orders and payments are working. I want to check the logs for myself. SRE’s diagnosis and suggested recovery. Suggested isn’t the same as confirmed, which is where I start digging. Get the historical window right The first query assumption I check is the date. This incident needs September 22, 17:30 to 19:45 UTC, not the last 24 hours. I use the Log Analytics workspace behind the demo and ask: Reconstruct the failure check for this historical window. Show the complete query and target before running it. Then break down the request routes and status codes. The reconstructed query and its saved result give me something concrete to inspect. This is my own reconstruction, not a replay of SRE’s query, so the counts differ a little from SRE’s snapshot. The query runbook includes the complete KQL, working Log Analytics API route, and scope for review. The result contains 120 gateway Orders 500 records, 115 gateway payments 502 records, corresponding backend failures, and 119 database connection timeouts. These are telemetry records, not distinct customer requests: the gateway and backend can record the same call. Exporter authentication errors and later route 404s are separate signatures, not more examples of the same database failure. Inspect the requests, dependencies, and exceptions Grouped counts tell me where to look. To understand the failure, I want to look at some of the actual log entries from that time window, not just the totals: Show the Orders requests, downstream dependencies, and database timeouts for this window. Where operation IDs are populated, follow the same operation. Tell me what is correlated and what is only nearby in time. The saved exception signature says the connection to the database endpoint timed out. That supports a connectivity problem; it doesn't independently prove the database was stopped. To connect a particular exception to a particular request, I need matching trace identifiers, not just nearby timestamps. The raw-record and correlation queries are the next checks, with their own execution approval. Widen my check beyond Orders An Orders-focused query is a useful starting point, not an impact assessment. I ask: Keep the same source and time window, but remove the Orders operation and dependency-target restrictions. What else has 5xx failures or the same database-connectivity signature? Keep unrelated errors separate. The saved broader result already includes payments, which SRE also identified. My goal is to verify that scope independently and see whether the wider query adds anything. I won't turn two affected paths into an all-services outage. This data also has blank application-role names, so I use the recorded operations and dependency targets rather than invent an application-name filter. Test the competing explanation A gateway deployment or connection setting could also cause trouble. I want to compare that explanation with the database diagnosis: Preview the historical deployment and configuration records needed for this window, plus independent database state or availability evidence. Do not use today's settings or health as proof of September 22. If the records aren't available, leave the check open. The first SRE record reports the database as stopped, but its Activity Log check found no stop event and may not have covered the right scope. Independent incident-window records would help distinguish stopped state from configuration or network problems. Current settings can't establish historical state, and missing or expired change history can't rule out a deployment. For now, the logs support connectivity trouble. Historical database state and the competing configuration explanation still need checking. The first record also doesn't confirm completed recovery; a later recurrence doesn’t tell us what happened on September 22.. Bring in evidence SRE may not have, or go where it cannot The logs got me to a connectivity problem. The next question is capacity: was something starved? That evidence lives in a Kusto cluster SRE Agent can’t reach, but I can. Private Connectors let me use my own authorized access from inside the same investigation. Copilot-private connections are separate from the SRE Agent connectors. Teams is ready for the update. For a performance or capacity question, I can choose an authorized connection, review the source and complete query, and approve its time range and filters before execution. Then I can compare the relevant measurements and units. This is an extension of the investigation. Granting capacity is a separate action with its own approval. I can do it from here, but it’s my call, not the agent’s. The connection stays private to Copilot. Connecting an SRE investigation doesn't give the agent my datasource permissions. If I want SRE to use a finding, I separately preview and approve the evidence to share. For checks I run locally, I use my identity and network access; a cloud-backed connector uses its provider's network path. Review the updated team message The team update should carry the refined root-cause analysis, including its limits: SRE's reported diagnosis, the independently observed failures, the scope, and the checks still open. Next prompt: Draft a September 22 update to send to the team in Teams. Include the reported diagnosis, observed Orders and payments failures, connectivity evidence, competing explanations, and remaining checks. Include the investigation link. Show me the exact message and destination before sending. That gives the team a useful starting point for the next check, with a link back to the investigation. Continue in the terminal I can carry the same investigation into a canvas-capable Copilot CLI session. I connect INC0922001 explicitly in that conversation; the App's connection doesn't carry over. Connect to INC0922001 from September 22. Give me a TL;DR that separates the reported diagnosis, independently checked evidence, and remaining questions. The terminal is another place where I like to work, and I still want to follow the same evidence and use my own context and tools. The SRE canvas provides tools to Copilot CLI too. The IDE and terminal guide covers a separate connection through the Azure MCP Server. So who stopped the database? We did, for the demo. The real production incident that inspired this was resource exhaustion, which is less satisfying, because every engineer wants it to be the database. It’s better when you can prove it. Try the canvas for your next investigation The useful loop isn't just reading an incident summary from an agent. It's taking SRE's starting diagnosis into Copilot, inspecting the evidence, testing alternatives, and bringing in authorized data from my own tools. Sometimes that confirms SRE Agent got it right. Sometimes it doesn’t. Either way, what I learn goes back into the agent’s memory so the next incident starts smarter. In this one, I learned we’re not resilient enough to this kind of failure yet, and we have hardening work to do. Install from Customize > Plugins > Awesome Copilot > Azure SRE Agent, start a new session, and ask: Open Azure SRE Agent canvas. Watch the demo video, bring an existing investigation, and tell us what you think. The canvas guide has setup details. Also see the October SRE Agent update and explore the other Azure canvases.Run an Ollaya decision model on Azure Container Apps
Many AI calls are not really conversations. A support message needs an intent. A workflow needs a route. A policy check needs a yes or no. An agent needs to select its next action from a known list. These are bounded decisions, but they are often sent to a general-purpose LLM. The LLM reads the request, generates an answer token by token, and then the application validates and parses the response. That is useful when the task needs reasoning or language generation. It is more machinery than necessary when the only valid answer is one of 60 labels. On September 15, 2026, TypeSafe released Jev, its first public System One Model, in early access. Jev has helped bring attention to models built specifically for decisions: structured state goes in, and typed choices with probabilities come out. Ollaya approaches the same problem from a self-hosted direction. It is a model runtime that can serve open decision models, including the winnow:e4b model used here. Jev and Ollaya are not connected products. Jev is a hosted model from TypeSafe; Ollaya provides a way to run decision models in infrastructure you control. They are related by the kind of work they target. What decision models are good at A decision model scores predefined options rather than generating open-ended text: Decision model Traditional LLM Selects from allowed choices Generates text Returns a probability for each decision Usually returns one generated answer Has a bounded, typed output Needs schema constraints and validation Supports confidence thresholds Often needs a separate confidence strategy Fits classification, routing, scoring, policy, and prioritization Fits generation, summarization, coding, and open-ended reasoning That narrower interface creates several practical benefits: Predictable outputs: the application receives an allowed value rather than text that must be repaired or parsed. Lower decision latency: there is no autoregressive output sequence to generate. Token savings: a decision can be returned without generating output tokens. Useful uncertainty: calibrated probabilities let the application act, reject, or escalate. Smaller infrastructure: a specialized model can fit on hardware that would be modest for a general LLM. Decision models are not replacements for every LLM call. They make sense when the possible outcomes are known before the request arrives. An LLM remains the better tool when the output itself is language, code, or open-ended reasoning. Why Azure Container Apps serverless GPU Azure Container Apps serverless GPUs make self-hosted inference feel much closer to consuming a managed API. You bring the container and model; Azure manages the underlying GPU infrastructure. The Consumption GPU profile provides: NVIDIA T4 or A100 GPUs without managing GPU nodes or a Kubernetes cluster. Automatic scaling with the option to scale to zero. Per-second GPU billing while replicas are running. Container Apps networking, identity, ingress, logging, and revision management. A private inference path where the model and request data stay inside your Azure environment. That last point matters for data-sensitive decisions. In the Ollaya-only configuration, text is sent to an internal Ollaya endpoint rather than an external model API. The model, API, persistent cache, and operational controls remain in the application's Azure environment. Serverless does not remove the need to think about cold starts. A model still needs to be loaded into VRAM. For low or sporadic traffic, scaling to zero can avoid idle GPU cost. For latency-sensitive traffic, a warm minimum replica avoids making a user wait for the model to load. Azure Files can retain the model layers across revisions so a new deployment does not download the full model again. The T4 is also an important part of this experiment. It is not the largest GPU available, but the goal is not to run the largest model. The goal is to match the hardware to a model designed for the task. A focused decision model on a T4 can compete with a hosted API when the workload is a bounded classification rather than text generation. The experiment I deployed winnow:e4b through Ollaya on an Azure Container Apps Consumption-GPU-NC8as-T4 workload profile. An authenticated API exposed the classifier while Ollaya remained on internal-only ingress. A second path sent the same requests to GPT-5.4 Nano through Azure OpenAI using managed identity. The evaluation used the complete 2,974-record test partition from Amazon MASSIVE 1.1. It contains 18 scenarios and 60 intents. Both providers received the same records, labels, and descriptions. The benchmark measured latency at concurrency 1 and throughput at concurrency 8. Providers and modes ran sequentially so one measurement did not load the service used by another. This was not a direct benchmark of Jev; it tested the same decision-model pattern with an open model that could run inside the Azure environment. What the results showed The benchmark ran on September 30, 2026. Provider Successful Intent accuracy Macro-F1 p50 p95 Winnow on T4 2,974 / 2,974 75.59% 76.40% 946 ms 960 ms GPT-5.4 Nano 2,974 / 2,974 79.12% 78.07% 1,369 ms 2,489 ms Nano led intent accuracy by 3.53 percentage points. Winnow was 30.9% faster at p50 and 61.4% faster at p95. This is the useful T4 result: a smaller GPU running the right specialized model matched and beat the hosted endpoint on response latency, though not on accuracy. At concurrency 8, Nano delivered 1.367 requests per second compared with Winnow's 1.120. Nano also had 25 requests fail after eight retries because the deployment exceeded its token-rate limit. Winnow completed all 2,974 requests, but its single loaded runner serialized work and increased queue time. The token comparison shows what the decision-only path removes: Provider, both benchmark passes Input tokens Cached input tokens Output tokens Winnow on T4 9,252,872 0 0 GPT-5.4 Nano 8,501,200 7,564,800 122,759 Winnow returned every decision without generating output tokens. Nano generated 122,759 output tokens across the latency and throughput passes. The rows do not represent equivalent billing models: Winnow consumes self-hosted GPU time, while Nano is metered by hosted token usage. The comparison isolates the generated tokens that a bounded decision did not need. Winnow also returned calibrated probabilities. At a 0.90 threshold, it accepted 53.73% of the records and was correct on 94.43% of those accepted decisions. An application could handle that high-confidence group locally and send only the uncertain remainder to an LLM or human reviewer. In brief A decision model is useful when software needs a bounded answer rather than generated language. In this experiment, Winnow on a serverless T4 traded some accuracy for lower latency, zero generated output tokens, private inference, and an explicit confidence signal. The practical design is often a combination: use the decision model for fast, high-confidence choices and reserve LLM calls for uncertain or open-ended work. Try it with the template The Azure Developer CLI template packages the Container Apps environment, T4 workload profile, authenticated API, internal Ollaya service, persistent model cache, GPU readiness checks, and benchmark. It supports two deployment modes: Mode What it deploys ollaya-only Private Winnow inference on a serverless T4 plus the authenticated API full The Ollaya deployment, GPT-5.4 Nano, and the comparison benchmark Read the deployment guide Inspect every benchmark prediction and retry Review the shared MASSIVE taxonomy
