New blog articles in Microsoft Community Hub

Partner Blog | Building the foundation for AI: Cloud, data, security, and AI skills for partners

JillArmourMicrosoft — Wed, 26 Aug 2026 21:08:26 GMT

Customers are moving beyond AI experimentation. They are looking for partners who can connect AI ambition to the cloud, data, security, governance, and business application capabilities required to put AI to work. That makes skilling across the Microsoft stack increasingly important.

It can also make the question of where to start more difficult. This month, there is a simpler starting point. The Microsoft Partner Skilling Hub Agent can recommend training, answer skilling-related questions, and generate personalized technical skilling plans based on your role and goals. From there, you can build a focused path across Frontier Transformation, agents, Microsoft 365 Copilot, certifications, hands-on learning, and co-sell execution.

The foundation for AI is broader than AI skills

Frontier Transformation is the shift from targeted AI pilots to repeatable, governed AI capabilities embedded into the flow of work, business processes, and customer engagement. For partners, delivering that transformation requires more than expertise in a single AI product. It requires teams that understand how cloud infrastructure, data, security, agents, and business applications work together.

That foundation matters across customer segments. For partners serving small and medium-sized businesses (SMBs), it can support you in guiding customers toward practical AI adoption while addressing security, governance, productivity, and business process needs together.

This month, focus your skilling plan on five areas: validating your technical expertise, building agent platform capabilities, developing AI business application skills, earning industry-recognized certifications, and applying those skills through hands-on learning.

Validate your Frontier Transformation expertise

The Frontier Transformation Engineer Badge gives technical professionals a path to validate their ability to design, build, and deliver AI solutions.

The journey progresses from Microsoft certifications through project-ready skilling and advanced Frontier engineering expertise. It is designed to build capability across agents, Microsoft 365 Copilot, Microsoft Foundry, Security, and the broader Frontier stack, giving your technical teams a structured way to move from foundational knowledge toward advanced implementation skills.

If you are developing a deeper AI engineering practice, consider identifying technical professionals who are ready to progress through the Frontier Transformation Engineer journey.

Continue reading here

Prepare for the launch of growth margins with API readiness resources

JillArmourMicrosoft — Wed, 26 Aug 2026 19:36:50 GMT

Starting October 1, 2026, growth margins will provide eligible partners with incremental margin on qualifying Microsoft 365 growth opportunities. Built on partner feedback, growth margins give you more flexibility to structure deals, compete for new business, reward reseller performance, and reinvest in capabilities that support long-term growth. Review the available readiness resources now so you can maximize growth opportunities, maintain business continuity, and prepare for upcoming technical changes.

Technical guidance is available to help you navigate API changes and implement best practices for data export and billing automation.

Review the partner resource collection.

Evaluate opportunities to incorporate growth margins into your Microsoft 365 sales and growth strategies.

Eligible partners may open an advisory case with Microsoft Partner Technical Consulting for API transition guidance.

Understanding and Reclaiming Storage After DELETE in Azure Database for PostgreSQL

Shweta_Anjankar — Wed, 26 Aug 2026 19:36:07 GMT

Why DELETE Does Not Immediately Shrink Storage and Azure Monitor Storage Graphs Continue to Show High Storage Consumption

PostgreSQL uses Multi-Version Concurrency Control, commonly called MVCC. MVCC allows readers and writers to work at the same time without blocking each other unnecessarily. This is great for concurrency, but it changes how storage behaves after DELETE and UPDATE operations.

When you execute DELETE:

DELETE FROM orders WHERE order_date < '2026-08-20';

PostgreSQL does not physically remove the rows from disk immediately.

Instead, it:

Marks the rows as dead tuples.
Keeps the row versions on disk so that transactions that started earlier can still see a consistent view of the data.
Leaves the space occupied by those rows in the table file.

As a result:

Logical row count decreases.
Physical table size typically remains unchanged.
Azure Monitor storage graphs continue to show similar storage consumption.

The deleted pages remain allocated until PostgreSQL rewrites or reorganizes them.

When you execute UPDATE:

UPDATE customers SET city = 'London' WHERE customer_id = 100;

PostgreSQL:

Marks the existing row version as obsolete.

Creates a new row version with the updated value.

Retains the old version until no active transaction needs it.

This behavior is a direct result of PostgreSQL's MVCC (Multi-Version Concurrency Control) architecture, which allows concurrent transactions without blocking readers.

This allows readers and writers to work concurrently, reduces locking, and gives long-running queries a consistent view of the data. The trade-off is that old row versions continue to occupy space until they can be cleaned up.

Learn more about MVCC PostgreSQL: Documentation: 7.1: Multi-Version Concurrency Control

Key Azure Metrics used for Storage analysis for Azure Database for PostgreSQL Flexible server:

The following metrics are usually the most useful during storage analysis:

Please refer Monitor Using Metrics and Logs in Azure Database for PostgreSQL Flexible Server - Azure Database for PostgreSQL | Microsoft Learn

After large DELETE operations, it's important to understand the difference between reclaiming database space for reuse and reducing Azure storage allocation and billing.

Storage That Can Be Reclaimed

The following types of space can often be reclaimed or reused:

Dead Tuple Space

Obsolete row versions created by UPDATE and DELETE are called dead tuples.

Dead tuples are no longer visible to normal queries, but they remain in the table files until vacuuming makes their space reusable. Workloads with frequent updates, deletes, batch processing, or ETL operations can accumulate large numbers of dead tuples. Autovacuum or manual VACUUM operations can clean up these tuples and make the space available for future database activity

How to Diagnose Table Bloat

Start by checking tables with high dead tuple counts:

SELECT relname, n_live_tup, n_dead_tup FROM pg_stat_user_tables ORDER BY n_dead_tup DESC;

Below query identifies the largest tables:

SELECT relname, pg_size_pretty(pg_total_relation_size(relid)) AS total_size FROM pg_catalog.pg_statio_user_tables ORDER BY pg_total_relation_size(relid) DESC;

Tables that are both large and have high dead tuple counts are strong candidates for maintenance review.

Table and Index Bloat

Over time, frequent INSERT, UPDATE, and DELETE operations can create bloat. This can be addressed using:

VACUUM FULL
pg_repack (preferred for production environments due to minimal locking)
Partition maintenance strategies

These operations can physically shrink tables and indexes, returning unused space to the operating system.

Temporary Storage Consumption

Storage consumed by temporary files generated from sorts, hash joins, and large query operations can be reclaimed automatically after the query completes. Identifying and tuning such workloads can prevent recurring storage spikes.

What Autovacuum Actually Does

PostgreSQL uses autovacuum to clean up obsolete row versions in the background. It also updates planner statistics and helps prevent transaction ID wraparound.

A regular vacuum operation:

Removes dead tuples when they are no longer visible to any transaction.
Marks their space as reusable.
Updates statistics when configured and also the maintenance information.
Does not normally shrink the table file.

Check whether Autovacuum is enabled:

SHOW Autovacuum;

Below SQL query reviews tables with high dead-tuple counts:

SELECT schemaname, relname, n_live_tup, n_dead_tup, last_autovacuum, last_vacuum FROM pg_stat_user_tables ORDER BY n_dead_tup DESC;

A high count may mean that autovacuum is not keeping up, large batch operations are producing dead tuples quickly, or old transactions are preventing cleanup.

Learn more about Autovacuum PostgreSQL: Documentation: 18: 19.10. Vacuuming

Following options helps us to reclaim/reuse the storage

Option 1: Standard VACUUM

VACUUM audit_logs;

Use standard VACUUM as routine maintenance. It makes deleted space reusable and helps improve database health, but it does not physically reduce table files.

Option 2: VACUUM FULL

To return unused table space to the operating system, PostgreSQL provides:

VACUUM FULL schema_name.table_name;

VACUUM FULL rewrites the table into a compact physical file containing only the active rows. It can substantially reduce the table's file size. However, it has several operational costs:

It takes an ACCESS EXCLUSIVE lock.
Reads and writes to the table are blocked.
It may run for a long time on large tables.
It requires additional temporary storage during the rewrite.
Associated indexes may also need time and space to be rebuilt.

Please refer VACUUM FULL - PostgreSQL wiki

Option 3: pg_repack

For production tables that cannot be unavailable for an extended period, pg_repack is often a better option.

Example:

pg_repack -d mydb -t schema_name.large_table

It rebuilds tables and indexes with much less blocking than VACUUM FULL. A short lock may still be required during setup or the final object swap, so it is not entirely lock-free.

Before using pg_repack on Azure Database for PostgreSQL, confirm that the required extension is supported and enabled for the server configuration.

Please refer https://learn.microsoft.com/azure/postgresql/troubleshoot/how-to-perform-fullvacuum-pg-repack

Option 4: Partitioning for Future Prevention

For historical data such as audit logs, telemetry, or event tables, partitioning is often the best long-term solution. Instead of deleting millions of old rows, you can remove an old partition.

DROP TABLE audit_logs_2026;

This approach is faster, produces less bloat, and reduces the pressure on Autovacuum.

Azure storage metrics may remain high after rows are deleted and vacuumed.

For example:

This can happen for two separate reasons:

PostgreSQL may have converted the deleted space into reusable space without shrinking its files.
Azure may retain the server's provisioned storage allocation even after PostgreSQL files are compacted.

As a result, a reduction in table size does not necessarily reduce the storage provisioned for the Flexible Server. Once Azure storage has grown, it generally should not be assumed that it will automatically shrink.

Provisioned Storage Cannot Be Reduced

Azure Database for PostgreSQL Flexible Server allows storage to be increased, but currently does not support reducing the configured storage size afterward.

For example:

Even if PostgreSQL reclaims internal space through VACUUM FULL or pg_repack, the server remains provisioned at 512 GB.

Storage Costs Continue Based on Provisioned Capacity

Because billing is tied to the provisioned storage allocation, deleting data does not automatically lower storage costs.

The following actions do not reduce provisioned storage billing:

DELETE operations
VACUUM
VACUUM FULL
Autovacuum cleanup
pg_repack

Best Practices to Avoid Storage Surprises

Do not run massive DELETE operations during peak workload hours.
Delete in controlled batches when a large purge is unavoidable.
Monitor dead tuples, table size, index size, WAL generation, and storage percent.
Investigate long-running transactions before assuming Autovacuum is broken.
Use partitioning for retention-based cleanup.
Use pg_repack or planned VACUUM FULL when physical compaction is required.
Set Azure Monitor alerts for storage utilization and plan capacity before full-disk scenarios occur.

Conclusion

When Azure PostgreSQL storage stays high after deleting, the database is usually behaving as designed. DELETE creates dead tuples and reusable internal space, but it does not automatically shrink physical files or reduce Azure allocated storage.

The right strategy depends on your goal. Use VACUUM for routine health, pg_repack for minimal-downtime compaction, VACUUM FULL when a maintenance window is acceptable, and partitioning to avoid large deletes in the first place.
Storage charges are tied to the amount of storage provisioned for the server. As a result, deleting data alone does not lower storage-related costs.

Your on-call rotation has a new member: 10 production incidents, end to end, with Azure SRE Agent

zmustafa — Wed, 26 Aug 2026 19:02:49 GMT

TL;DR
1. Why the "investigation" half is the real prize
2. What Azure SRE Agent actually does
3. Anatomy of an agent-run incident
4. Build the demo lab
5. Set your guardrails before your first incident
6. The ten use cases
7. The ITSM integration model
8. Reality check: where you have to build
9. Approval and autonomy policy
10. Cross-cutting security controls
- What the platform gives you for free
- The audit trail
11. A 30/60/90 pilot that survives contact with your CAB
- The best first five candidates
- What to measure in week one
12. Measuring whether it's actually working
13. Resources
Closing

What this post is. A hands-on, reproducible walkthrough of ten real production failure modes — App Service, AKS, Azure SQL, Cosmos DB, VMs, VM Scale Sets, Application Gateway, and Service Bus — each one driven end to end by Azure SRE Agent: detection, hypothesis-driven investigation, ITSM ticketing, bounded remediation, recovery validation, and follow-up records.

What this post is not. A claim that you install SRE Agent and all of this happens on day one. Every workflow below needs telemetry, scoped RBAC, a response plan, an approved action surface, and an ITSM integration. I'll be explicit about which parts are documented product behavior and which parts you have to wire yourself — because that distinction is the difference between a demo that works on stage and one that works at 3 AM.

TL;DR

The pattern	Alert → agent investigates read-only → agent opens/updates the ticket with evidence → agent proposes a bounded action → human approves → agent executes → agent validates recovery → agent files the follow-up record
The unlock	Not "AI fixes prod." The unlock is that investigation — the 20 minutes of tab-switching between Azure Monitor, App Insights, deployment history, and Activity Log — is fully automated and consistent, and the fix arrives pre-justified with an evidence chain
The guardrail	Read-only automatic. Ticketing automatic. Production writes in Review mode. Guest-OS work through fixed-purpose runbooks, never a shell prompt
The reality check	SRE Agent hard-blocks `delete`/`remove` and all `az keyvault` commands, respects Azure management locks, and only one incident platform can be active at a time. Several patterns in this post need a custom tool or MCP server to complete the loop
Time to first value	One resource group, one alert rule, one response plan. You can reproduce use case #1 in an afternoon

1. Why the "investigation" half is the real prize

Every conversation about AI in operations goes straight to remediation. "Will it restart my app?" That's the least interesting question, and it's the one with the most downside risk.

Think about what actually consumes the minutes during a Sev1. The alert fires. Someone acknowledges. Then:

Open Azure Monitor, confirm the metric is real and not a probe artifact
Open Application Insights, find the dominant exception
Open the deployment pipeline, find what shipped and when
Open Activity Log, check whether someone changed configuration
Open Resource Health, rule out a platform incident
Open the other environment, confirm the previous version is healthy
Assemble all of that into a sentence a human can act on

That's twenty minutes of context assembly performed by a tired human, differently every time, with quality that depends entirely on who happens to be on call. It is the single most automatable part of incident response and the part nobody automates, because scripts can't reason about which of six hypotheses fits the evidence.

This is exactly what root cause analysis in SRE Agent is designed for. The agent doesn't grep logs — it forms hypotheses and invalidates them:

HYPOTHESIS 1: Recent deployment broke something ├─ Checked: Last deployment was 3 days ago ├─ Evidence: Error rate stable until 30 minutes ago └─ Result: INVALIDATED HYPOTHESIS 2: Database overloaded ├─ Checked: Azure SQL metrics (CPU, DTU, connections) ├─ Evidence: DTU at 98%, query duration 4x normal ├─ Traced: SELECT * FROM orders WHERE... taking 8.2s └─ Result: VALIDATED ROOT CAUSE: Orders table missing index on customer_id column. Query plan shows full table scan on 2.1M rows. RECOMMENDED ACTION: Add index on orders.customer_id Similar fix applied in INC-2341 (3 weeks ago)

That last line — recalling a similar incident from three weeks ago — is the compounding part. Every thread produces a session insight: symptoms observed, steps that worked, root cause, and pitfalls to avoid. Thirty minutes after a thread goes quiet, the agent indexes those learnings. Next time the same resource misbehaves, that history surfaces first.

So the framing for the rest of this post: remediation is the punchline, but investigation is the product. Every one of the ten use cases below has a large read-only phase you can turn on tomorrow with zero write permissions, and a small write phase you should gate behind approval for a long time.

2. What Azure SRE Agent actually does

Before the use cases, here is the honest capability map, drawn from the product documentation rather than from a keynote.

The three primary use cases

Use case	What it means
Automate incidents	Alert fires → agent queries monitoring tools, correlates signals across systems, identifies probable root cause, proposes mitigations
Automate scheduled workflows	Proactive health checks, compliance sweeps, and routine tasks on a schedule, with results routed to your incident platform or notification channel
Investigate and advise	Natural-language questions — "what changed in the last hour?" — answered with grounded citations

The five extension primitives

Everything you customize sits in one of five buckets:

Primitive	What it is	When you reach for it
Skills	Procedural guidance (`SKILL.md`) plus optional attached tools; auto-loaded when relevant	Team troubleshooting runbooks that should also execute
Subagents / custom agents	Purpose-built specialists invoked via `/agent` or routed to by a response plan	A `DatabaseExpert` that owns every SQL incident
Python tools	Custom logic, transformations, API calls	Anything that needs code, e.g. writing to the ServiceNow Table API
MCP servers	40+ managed connectors (Datadog, New Relic, Splunk, Elastic, Dynatrace…) plus any custom MCP tool	Bringing your non-Azure telemetry and your ITSM write surface into the loop
Agent hooks	Event-triggered automations at `Stop` and `PostToolUse`	Policy enforcement, audit emission, blocking risky commands

Six generic subagents ship built in — Explore, Plan, CodeReview, Bash, Verification, GeneralPurpose — and the agent can parallelize investigation, planning, review, shell, and verification work across them.

A permission gate sits in front of all five primitives and evaluates every proposed tool call before it runs.

Integrations you can assume

Category	Supported
Monitoring	Azure Monitor (metrics, logs, alerts, workbooks), Application Insights, Log Analytics, Grafana
Incident platforms	Azure Monitor Alerts, PagerDuty, ServiceNow — only one active at a time
Source control / CI	GitHub (repos, issues), Azure DevOps (repos, work items)
Data	Azure Data Explorer (Kusto), MCP servers
Comms	Slack, Microsoft Teams

⚠️ Design constraint worth internalizing early. Only one incident platform can be active at a time, and switching disconnects the current one. If your org runs PagerDuty for paging and ServiceNow for records of truth, you must pick which one the agent is bound to and reach the other through a connector or custom tool. Every use case below assumes ServiceNow is the bound platform.

Skills vs. custom agents vs. knowledge files

The three are constantly confused. This table settles it:

	Skills	Custom agents	Knowledge files
Access	Automatic when relevant	Explicit (`/agent`) or routed by response plan	Automatic search
Tools	Can attach	Has its own	None
Context	Uses thread context	Shares thread context (no clean slate)	Reference only
Best for	Team procedures with execution	Domain specialists	Runbooks, architecture docs

Practical limits: a maximum of five concurrent active skills (oldest auto-unloads), knowledge base uploads up to 16 MB per file, and custom agent knowledge bases up to 1,000 files.

3. Anatomy of an agent-run incident

Every use case in section 6 is an instance of this one shape. Learn it once.

Five properties make this shape safe, and they're worth stating as rules:

The read phase has no blast radius. Turn it on everywhere, immediately, with Reader.
One action per incident. Not "roll back and scale and restart." One bounded, reversible step, then re-measure.
The proposed action is always the smallest reversible one. Swap a slot, don't redeploy. Bump one service tier, don't resize the cluster.
Validation is a first-class phase, not a vibe. Define the metric, the threshold, and the duration before you approve.
Technical recovery ≠ business recovery. Use case #10 makes this painfully clear.

4. Build the demo lab

Everything below runs in a single throwaway resource group. Nothing here should touch a subscription you care about.

🧪 Lab hygiene. Create it, demo it, delete it. az group delete -n rg-sre-agent-demo --yes --no-wait when you're done. Several of these use cases deliberately break things.

4.1 Resource group and agent

LOC=eastus2 RG=rg-sre-agent-demo az group create -n $RG -l $LOC

Create the SRE Agent from the Azure portal and point it at $RG. Three things are created for you automatically:

an Application Insights instance (this is where your audit trail lands),
a Log Analytics workspace,
a user-assigned managed identity (UAMI) — the identity every action runs as.

4.2 Choose a permission level deliberately

At creation you pick one of two levels:

Level	Grants	Use it when
Reader	Core monitoring roles + resource-type reader roles. Prompts for temporary elevation via on-behalf-of when it needs to act	Start here. Production. Weeks 1–4 of any pilot
Privileged	Core monitoring roles + resource-type contributor roles	Non-production, or after a proven pilot

Regardless of level, these are always assigned:

Role	Scope	Why
`Reader`	Resource group	See resources and properties
`Log Analytics Reader`	Resource group	Query logs and workspaces
`Monitoring Reader`	Resource group	Read metrics
`Monitoring Contributor`	Subscription	Acknowledge and close Azure Monitor alerts

Grant additional access explicitly and narrowly:

SUB=$(az account show --query id -o tsv) AGENT_MI=<agent-managed-identity-principal-id> # Read everywhere you want visibility az role assignment create \ --assignee $AGENT_MI \ --role "Reader" \ --scope "/subscriptions/$SUB" # Write ONLY where you intend the agent to act az role assignment create \ --assignee $AGENT_MI \ --role "Website Contributor" \ --scope "/subscriptions/$SUB/resourceGroups/$RG/providers/Microsoft.Web/sites/app-checkout-demo"

📌 A sharp edge. You can't remove individual permissions from an agent — only entire resource groups. Removing a resource group from the agent's scope revokes all access to it. Plan your resource group boundaries as your blast-radius boundaries, because that's exactly what they are.

4.3 Connect ServiceNow

Use a dedicated, least-privileged integration user — not a shared admin account.

Method	When	What you need
Basic auth	Quick setup, PDI, testing	Username + password, `itil` or `admin` role
OAuth 2.0	Production	ServiceNow OAuth app (client ID + secret); register redirect `https://logic-apis-{region}.consent.azure-apim.net/redirect`

Scope the connection so you don't drown:

Assignment group — essential on a shared enterprise instance
Priority — Critical through Planning
Category

Scanner defaults worth knowing:

Setting	Value
Scan interval	1 minute
Incidents per page	20
Max incidents per cycle	220 (11 pages)
Initial lookback	30 days

Setup performs a real connectivity check by fetching an actual incident, so credential and endpoint mistakes surface immediately instead of six hours later when nothing syncs.

🚨 Delete the quickstart plan. Connecting an incident platform auto-creates a quickstart_handler response plan that runs in fully autonomous mode across all impacted services. If you then build your own plans, incidents get routed twice or to the wrong agent. Go to Builder → Incident response plans → Table view and delete it before you do anything else.

4.4 Connect deployment correlation

Half the use cases below hinge on "what shipped three minutes before the spike." That correlation requires a source control connector:

GitHub — repositories and issues
Azure DevOps — repos and work items

Connect one. Without it, the agent can still read Azure Activity Log and deployment history, but it can't reach commits, PRs, or work items.

4.5 Create the demo resources

# 1 · App Service with a staging slot (use cases 1) az appservice plan create -g $RG -n plan-demo --sku P1V3 --is-linux az webapp create -g $RG -p plan-demo -n app-checkout-demo --runtime "DOTNETCORE:8.0" az webapp deployment slot create -g $RG -n app-checkout-demo --slot previous # 2 · AKS (use case 2) az aks create -g $RG -n aks-demo --node-count 3 --generate-ssh-keys --enable-addons monitoring # 3 · Azure SQL (use case 3) az sql server create -g $RG -n sqlsrv-sre-demo -u sqladmin -p "<use-a-generated-password>" az sql db create -g $RG -s sqlsrv-sre-demo -n db-customer --service-objective S1 # 4 · Cosmos DB (use case 4) az cosmosdb create -g $RG -n cosmos-sre-demo az cosmosdb sql database create -g $RG -a cosmos-sre-demo -n catalog az cosmosdb sql container create -g $RG -a cosmos-sre-demo -d catalog \ -n products --partition-key-path /category --throughput 400 # 5–7 · VMs (use cases 5, 6, 7) az vm create -g $RG -n vm-payments-linux --image Ubuntu2204 --size Standard_B2s --generate-ssh-keys az vm create -g $RG -n vm-claims-win --image Win2022Datacenter --size Standard_B2ms \ --admin-username azureadmin --admin-password "<use-a-generated-password>" # 8 · VM Scale Set (use case 8) az vmss create -g $RG -n vmss-api-demo --image Ubuntu2204 --instance-count 12 \ --upgrade-policy-mode automatic --generate-ssh-keys # 10 · Service Bus (use case 10) az servicebus namespace create -g $RG -n sb-sre-demo --sku Standard az servicebus queue create -g $RG --namespace-name sb-sre-demo -n order-events \ --enable-dead-lettering-on-message-expiration true

Install the Azure Monitor Agent on the VMs and associate a data collection rule — without guest telemetry, use cases 5, 6, and 7 have nothing to detect.

4.6 One custom agent per domain

Don't build a single mega-agent. Build specialists and let response plans route to them:

Custom agent	Owns	Attached tools
`DeploymentAnalyzer`	App Service, AKS, anything release-correlated	`RunAzCliReadCommands`, GitHub connector, Kusto
`DatabaseExpert`	Azure SQL, Cosmos DB	`RunAzCliReadCommands`, Kusto, read-only SQL diagnostics tool
`GuestOSResponder`	VM / VMSS / IIS	`RunAzCliReadCommands`, fixed-purpose runbook tools only
`NetworkPathExpert`	Application Gateway, NSG, DNS, TLS	`RunAzCliReadCommands`
`IntegrationExpert`	Service Bus, Event Hubs	`RunAzCliReadCommands`, Kusto

A custom agent definition is small:

name: database_expert system_prompt: | You are a database specialist for this estate. Analyze query performance, diagnose connection and saturation issues, and recommend the smallest reversible mitigation. Never propose schema changes, index changes, plan forcing, or session termination — those are DBA-owned and require a change record. handoff_description: Handles Azure SQL and Cosmos DB troubleshooting tools: - execute_kusto_query - RunAzCliReadCommands allowed_skills: - azure-sql-saturation-runbook - cosmos-throughput-runbook

Note what that system_prompt is really doing: it is narrowing the action space. Half of production safety with an agent is telling it, in plain English, which categories of fix are off the table.

4.7 Response plans

Create one plan per domain, all starting in Review:

Plan	Filter	Custom agent	Mode
`appsvc-p1`	Priority 1 + 2, service `checkout`	`DeploymentAnalyzer`	Review
`aks-p1`	Priority 1, service `orders-api`	`DeploymentAnalyzer`	Review
`db-critical`	Priority 1 + 2, service `customer-api`	`DatabaseExpert`	Review
`vm-guest`	Priority 1 + 2, title contains `disk`, `memory`, `CPU`	`GuestOSResponder`	Review
`network-p1`	Priority 1, title contains `502`, `backend`	`NetworkPathExpert`	Review
`integration-p1`	Priority 1, service `fulfillment`	`IntegrationExpert`	Review

Filters available: severity/priority (multiselect), impacted service, incident type, and title contains. Plans can be turned off without deleting them, which is exactly what you want during maintenance windows.

5. Set your guardrails before your first incident

If you read only one section of this post, read this one. The guardrails are not paperwork; they are the reason this is deployable.

5.1 Run modes

Mode	Behavior	Default
Review	Agent proposes; you approve or deny	Agent-level default
Autonomous	Agent executes immediately and reports	Per-plan and per-task default

Two subtleties that bite people:

Run modes are set per response plan and per scheduled task, not globally. The agent-level setting is only a fallback. And per-plan the default is Autonomous — so if you don't set it, you get autonomy.
Review mode shows Approve/Deny only for Azure infrastructure operations (Azure CLI, ARM writes). Sending an email, posting to Teams, or querying an external source proceeds based on the agent's reasoning. To gate those, you need hooks or tool access policies.

Only users holding the SRE Agent Administrator role can approve. Standard users cannot, and personal Microsoft accounts can't authorize on-behalf-of at all — it requires a work or school (Entra ID) account.

5.2 What the product blocks for you

These are enforced at the command level, independent of your RBAC:

Guardrail	Behavior
Delete operations	The agent never runs `delete` or `remove` commands. It returns an error pointing you at the portal
Key Vault	All `az keyvault` commands are blocked, to prevent credential exposure
Management locks	Resources with `ReadOnly` locks can't be modified, regardless of permissions or run mode
Subscription validation	Subscription IDs are validated as well-formed GUIDs before execution

💡 Use the delete block architecturally. Put a ReadOnly management lock on anything that must never change during an incident — your Key Vaults, your production databases, your golden images. That lock is respected before every modification, which gives you a control that survives RBAC drift and misconfigured response plans.

5.3 Hooks: the guardrail you write yourself

Two events are supported — Stop (agent about to return a final response) and PostToolUse (a tool finished). Hooks run as an LLM prompt or a sandboxed command script.

Here is the single most useful hook for this entire post — a deterministic policy gate on shell execution:

hooks: PostToolUse: - type: command matcher: "Bash|ExecuteShellCommand" timeout: 30 failMode: block script: | #!/usr/bin/env python3 import sys, json, re context = json.load(sys.stdin) command = context.get('tool_input', {}).get('command', '') dangerous = [ r'\brm\s+-rf\b', r'\bsudo\b', r'\bchmod\s+777\b', r'\bmkfs\b', r'\bdd\s+if=', r'\btruncate\b', r'\bDROP\s+TABLE\b', ] for pattern in dangerous: if re.search(pattern, command, re.IGNORECASE): print(json.dumps({"decision": "block", "reason": f"Blocked by policy: {pattern}"})) sys.exit(0) print(json.dumps({"decision": "allow"}))

And a Stop hook that refuses to let the agent declare victory without evidence:

hooks: Stop: - type: prompt model: ReasoningFast timeout: 30 prompt: | Review the agent's final response. $ARGUMENTS It is only acceptable if it contains ALL of: 1. The specific resource acted upon (full name or resource ID) 2. Metric values BEFORE and AFTER the action 3. The observation window over which recovery was confirmed 4. Who approved the action and at what UTC timestamp Respond with: {"ok": true} {"ok": false, "reason": "<what is missing>"}

Hook mechanics you'll want on hand:

Setting	Default	Range / notes
`timeout`	30s	1–300s
`failMode`	`allow`	`allow` or `block`
`maxRejections`	3	1–25; prompt-type `Stop` hooks only
`matcher`	—	Regex, anchored `^(pattern)$`, case-sensitive; `*` matches all
Script size	—	64 KB max
Shebangs	—	`#!/bin/bash`, `#!/usr/bin/env python3`

⚠️ The footgun: for Stop hooks, a rejection without a reason field is treated as approval. Always populate reason.

Agent-level hooks and custom-agent-level hooks both run when both match; agent-level fires first.

5.4 The recommended production policy

This is the policy I'd put in front of a change board:

Allow read-only investigation automatically, everywhere.
Allow automatic incident creation and work-note updates — after sanitization.
Use Review mode for all production remediation.
Require human approval for every production write.
Use fixed-purpose runbooks instead of unrestricted VM shell access.
Require separate approval for destructive or data-affecting actions.
Initially require human confirmation before resolving an incident.

6. The ten use cases

Each use case follows the same seven-part structure so you can skim to the one you're firefighting:

What happened → How Azure Monitor detected it → How the agent found the cause → Agent action table → Recovery note → Try it yourself → Guardrails

In the action tables, Type is one of:

Type	Meaning
`Read`	No blast radius. Safe to run automatically
`Decision`	Agent reasoning or an approval gate. No resource change
`Write (ITSM)`	Ticket create/update. Sanitized
`Write (Azure)`	Azure control-plane change. Approval required in production
`Write (Guest OS)`	Inside the VM, via a fixed-purpose runbook. Approval required
`Write (K8s/DevOps)`	Cluster or pipeline change. Approval required
`Validation`	Post-action measurement against defined thresholds

Use case #1 — App Service: HTTP 500 after deployment

The one you should build first. Clean trigger, bounded action, unambiguous validation.

What happened

Release 2026.03.12.4 was deployed to a production checkout App Service. The new code referenced an application setting that was never defined in the production slot. The application started fine — which is what makes this class of failure nasty — but every checkout operation that touched that setting returned HTTP 500.

How Azure Monitor detected it

Azure Monitor and Application Insights fired on a composite condition, not a single metric:

HTTP 5xx rate above the operational threshold
Failed availability tests
Increased application exceptions
Increased dependency failures
Request latency above baseline

P1 – Checkout App Service returning elevated HTTP 500 responses

How the agent found the cause

Identified the affected App Service and slot.
Queried HTTP response and latency metrics.
Queried Application Insights failed requests and exceptions.
Identified the missing-setting exception as the dominant failure.
Reviewed deployment history and Azure Activity Log.
Correlated the error increase with release 2026.03.12.4.
Compared production against the previous deployment slot.
Confirmed the previous slot remained healthy.
Confirmed no matching Azure platform incident.

Agent finding:

HTTP 500 responses increased from 0.4% to 18% within three minutes of release 2026.03.12.4. Most failures reference a missing application setting. The previous deployment slot passes availability and dependency checks.

Notice the shape of that sentence: a delta, a time correlation, and a known-good comparison. That's what makes it actionable rather than merely true.

Agent action table

#	Action	Type	Detail
1	Investigate alert	Read	Reads HTTP 5xx rate, failed requests, latency, availability-test results
2	Analyze exceptions	Read	Identifies missing-configuration exception as dominant failure
3	Correlate deployment	Read	Finds release 2026.03.12.4 deployed three minutes before the spike
4	Compare slots	Read	Confirms previous slot healthy while production is failing
5	Rule out platform issue	Read	Checks Azure Resource Health and dependency health
6	Assess blast radius	Read	Determines checkout affected, unrelated services healthy
7	Create SNOW INC	Write (ITSM)	P1 against the Checkout CI, assigned to Application Operations
8	Update INC evidence	Write (ITSM)	Adds exceptions, deployment correlation, resource links, affected operations
9	Classify response	Decision	Classifies as deployment-caused; selects rollback
10	Pause release	Write (DevOps)	Pauses the failed release pipeline
11	Request approval	Decision	Requests approval to swap the previous healthy slot into production
12	Swap slot	Write (Azure)	Performs the approved swap only on the named App Service
13	Validate recovery	Validation	Confirms 5xx, latency, exceptions, dependencies, availability recover
14	Update/resolve INC	Write (ITSM)	Records approval, rollback, timestamps, recovery evidence
15	Create SNOW PRB	Write (ITSM)	Problem record for configuration-validation improvements

Recovery note

Production was reverted to the previous healthy deployment slot at 14:26 UTC. HTTP 500 responses declined from 18% to below 1%, and availability tests passed for 15 consecutive minutes. Preliminary cause: missing production configuration in release 2026.03.12.4.

Try it yourself

Break it:

# Put a healthy build in the 'previous' slot first, then break production # by deploying code that reads an app setting which only exists in staging. az webapp config appsettings delete -g $RG -n app-checkout-demo \ --setting-names Checkout__PaymentProviderKey az webapp restart -g $RG -n app-checkout-demo

Alert it:

az monitor metrics alert create -g $RG -n "alert-checkout-5xx" \ --scopes "/subscriptions/$SUB/resourceGroups/$RG/providers/Microsoft.Web/sites/app-checkout-demo" \ --condition "total Http5xx > 20" \ --window-size 5m --evaluation-frequency 1m --severity 1 \ --description "P1 – Checkout App Service returning elevated HTTP 500 responses"

Ask it (before the alert, to see the read phase in isolation):

app-checkout-demo is returning HTTP 500s. Do not change anything. Investigate and tell me: 1. the dominant exception and its share of total failures 2. what deployed in the 30 minutes before the error rate changed 3. whether the 'previous' slot is healthy right now 4. whether Azure Resource Health shows a platform issue Give me the evidence chain and the smallest reversible mitigation.

Expect it to propose:

Proposed action: Swap slot 'previous' into production on app-checkout-demo Risk: brief connection drain (~10s). Fully reversible by swapping back. Validation: Http5xx < 1% and availability test passing for 15 minutes. [Approve] [Deny]

Verify it:

requests | where timestamp > ago(1h) | summarize failed = countif(success == false), total = count() by bin(timestamp, 1m) | extend failureRate = 100.0 * failed / total | render timechart

🎬 Best thing to record for a demo: the moment the Approve button appears with the evidence already attached. That single screen is the whole value proposition.

Guardrails

Scope write permissions to the named App Service, not the resource group.
Verify the previous slot is healthy before swapping — a swap into a broken slot doubles the outage.
Require human approval for production slot swaps.
Preserve the failed deployment for analysis; don't let the pipeline overwrite it.
Never grant subscription-level Contributor or Owner.

Use case #2 — AKS: pods in CrashLoopBackOff

What happened

Image orders-api:4.18.0 referenced a configuration key that didn't exist in the production namespace. Eight of ten pods entered CrashLoopBackOff. The two survivors couldn't carry production traffic, producing latency and ingress 5xx.

How Azure Monitor detected it

Managed Prometheus / Container Insights detected:

Unavailable replicas
Increasing container restart count
CrashLoopBackOff pod state
Failed readiness and liveness checks
Elevated ingress HTTP 5xx
Reduced successful-request rate

P1 – Orders API unavailable replicas in production AKS

How the agent found the cause

Identified the cluster, namespace, and deployment.
Reviewed deployment availability and pod states.
Examined logs from failing containers.
Reviewed Kubernetes warning events.
Compared current and previous ReplicaSets.
Correlated the failure with revision 42.
Identified the missing configuration key.
Confirmed revision 41 was previously healthy.
Checked node CPU, memory, storage, networking, status.
Determined AKS infrastructure was healthy.

Agent finding:

Eight of ten Orders API pods entered CrashLoopBackOff during rollout of revision 42. Container logs show a missing configuration key. Revision 41, using image 4.17.6, was healthy.

Step 10 is the one people skip. "Rule out the infrastructure" is what stops you from spending an hour on a node pool that was never the problem.

Agent action table

#	Action	Type	Detail
1	Investigate workload	Read	Deployment status, unavailable replicas, readiness, restart counts
2	Analyze pod logs	Read	Startup failure from a missing configuration key
3	Analyze events	Read	Image pulls, mounts, scheduling, probes, container events
4	Correlate rollout	Read	Revision 42 deployed immediately before failure
5	Compare ReplicaSets	Read	Revision 41 was the previous healthy workload
6	Rule out infrastructure	Read	Nodes, memory, CPU, networking, storage healthy
7	Assess blast radius	Read	Eight of ten replicas unavailable
8	Create SNOW INC	Write (ITSM)	P1 against the Orders API CI
9	Update INC evidence	Write (ITSM)	Namespace, image, revision, errors, rollout correlation
10	Classify response	Decision	Application/configuration-caused
11	Request approval	Decision	Approval to pause and roll back revision 42
12	Pause failed rollout	Write (K8s/DevOps)	Prevents the release progressing or being reapplied
13	Roll back deployment	Write (K8s)	Rolls back only the named deployment to revision 41
14	Validate replicas	Validation	All expected replicas ready and stable
15	Validate application	Validation	Ingress errors decline; synthetic orders succeed
16	Update/resolve INC	Write (ITSM)	Revision, approval, rollback, recovery evidence
17	Create defect/task	Write (ITSM)	Add configuration validation to CI/CD

Recovery note

Orders API was rolled back from revision 42 to revision 41. All ten replicas are ready, restart counts have stabilized, ingress HTTP 5xx responses are below threshold, and synthetic order transactions are succeeding.

Try it yourself

Break it:

az aks get-credentials -g $RG -n aks-demo kubectl create namespace orders kubectl create deployment orders-api -n orders --image=nginx:1.25 --replicas=10 kubectl rollout status deployment/orders-api -n orders # revision 41 equivalent, healthy # Now break it: point at an image whose entrypoint requires a missing env var kubectl set image deployment/orders-api -n orders orders-api=busybox:1.36 kubectl patch deployment orders-api -n orders --type=json -p='[ {"op":"add","path":"/spec/template/spec/containers/0/command", "value":["sh","-c","test -n \"$ORDERS_CONFIG_KEY\" || (echo \"FATAL: missing ORDERS_CONFIG_KEY\" >&2; exit 1); sleep 3600"]} ]'

Watch it break:

kubectl get pods -n orders -w kubectl rollout history deployment/orders-api -n orders

Ask it:

Pods in namespace 'orders' on aks-demo are crash looping. Read-only. Tell me: which revision introduced it, the exact container error, whether the node pool is healthy, and how many replicas are actually serving. Then tell me the last known-good revision and why you believe it was healthy.

Expect it to propose: kubectl rollout undo deployment/orders-api -n orders --to-revision=<n> scoped to that single deployment.

Verify it:

kubectl get deployment orders-api -n orders \ -o jsonpath='{.status.readyReplicas}/{.status.replicas}{"\n"}' kubectl get pods -n orders --no-headers | awk '{print $4}' | sort | uniq -c

Guardrails

Restrict access to the required cluster, namespace, and deployment.
Do not grant cluster-admin.
Require approval for rollback.
Coordinate with GitOps reconciliation. If Flux or Argo owns that deployment, a kubectl rollout undo gets reverted within minutes and you've created a flapping outage. Either pause reconciliation first or roll back through Git.
Do not permit namespace, persistent-volume, or cluster deletion.

Use case #3 — Azure SQL Database: CPU saturation and timeouts

The use case where the agent's job is to buy time, not to fix the problem.

What happened

A query execution plan changed after an application release. The new plan consumed substantially more CPU and workers. The database saturated, producing SQL dependency timeouts and failed customer requests.

How Azure Monitor detected it

Sustained CPU saturation
Worker or session pressure
Increased connection failures
SQL dependency timeouts
Query-duration deviation from baseline
Degraded customer API success rate

P1 – Customer API database saturation causing request timeouts

How the agent found the cause

Reviewed database CPU, workers, sessions, connections.
Reviewed application SQL dependency failures.
Checked for blocking and deadlocks.
Ran approved read-only Query Store diagnostics.
Identified the primary CPU-consuming query.
Detected a recent execution-plan change.
Correlated with an application release.
Ruled out Azure service health and storage issues.
Determined a temporary scale operation could restore service.
Left permanent query remediation to the DBA team.

Agent finding:

Database CPU has remained saturated for 17 minutes. One query accounts for most recent CPU consumption and changed execution plan shortly before the incident. Application SQL dependency timeout rate is 23%.

Step 10 is the design decision that makes this safe. The agent correctly diagnoses a plan regression and then deliberately does not fix it, because forcing a plan or dropping an index is a permanent, DBA-owned, change-controlled action. It buys capacity instead.

Agent action table

#	Action	Type	Detail
1	Investigate database	Read	CPU, workers, sessions, connections, storage, availability
2	Analyze app impact	Read	SQL dependency latency, failures, affected API operations
3	Check blocking	Read	Approved diagnostics for blocking, deadlocks, connection growth
4	Analyze Query Store	Read	Highest-impact query and recent plan change
5	Correlate changes	Read	Recent application and database deployments
6	Rule out platform issue	Read	Service health, storage, database availability
7	Assess blast radius	Read	Affected applications; other databases healthy
8	Create SNOW INC	Write (ITSM)	P1 against the production database CI
9	Update INC evidence	Write (ITSM)	Utilization, query ID, timeouts, change correlation
10	Classify response	Decision	Scaling is mitigation; query changes remain DBA-owned
11	Calculate bounded scale	Read	Selects the smallest pre-approved capacity increase
12	Request approval	Decision	Database owner or incident commander
13	Scale database	Write (Azure)	Increases the affected database by one approved service step
14	Validate recovery	Validation	CPU, workers, timeouts, application success recover
15	Update INC	Write (ITSM)	Previous/new capacity, approval, timing, results
16	Create SNOW PRB	Write (ITSM)	Permanent query-remediation work
17	Create scale-down task	Write (ITSM)	Task/change to restore normal capacity after stability

Recovery note

Azure SQL capacity was temporarily increased by one approved service step. CPU declined from sustained saturation to 54%, and SQL dependency timeouts returned to baseline. Query Store indicates a probable execution-plan regression requiring permanent DBA remediation.

Try it yourself

Break it — generate a saturating workload against db-customer (an unindexed LIKE '%...%' scan in a tight loop from a container in the same region works fine on an S1).

Alert it:

az monitor metrics alert create -g $RG -n "alert-sql-cpu" \ --scopes "/subscriptions/$SUB/resourceGroups/$RG/providers/Microsoft.Sql/servers/sqlsrv-sre-demo/databases/db-customer" \ --condition "avg cpu_percent > 90" \ --window-size 5m --evaluation-frequency 1m --severity 1 \ --description "P1 – Customer API database saturation causing request timeouts"

Ask it:

db-customer is saturated. Read-only investigation. Identify the top CPU-consuming query, whether its plan changed recently, and what application release correlates. Do NOT propose index, schema, plan-forcing, or session-kill actions. Propose only the smallest temporary capacity step that restores service, and tell me what it costs per day and when we should scale back down.

Read-only Query Store diagnostics the agent should run:

SELECT TOP 10 qsq.query_id, qsp.plan_id, qsp.last_execution_time, SUM(qsrs.count_executions) AS executions, SUM(qsrs.avg_cpu_time * qsrs.count_executions) AS total_cpu_us FROM sys.query_store_query AS qsq JOIN sys.query_store_plan AS qsp ON qsp.query_id = qsq.query_id JOIN sys.query_store_runtime_stats AS qsrs ON qsrs.plan_id = qsp.plan_id JOIN sys.query_store_runtime_stats_interval AS qsrsi ON qsrsi.runtime_stats_interval_id = qsrs.runtime_stats_interval_id WHERE qsrsi.start_time > DATEADD(hour, -2, GETUTCDATE()) GROUP BY qsq.query_id, qsp.plan_id, qsp.last_execution_time ORDER BY total_cpu_us DESC;

Expect it to propose: az sql db update -g $RG -s sqlsrv-sre-demo -n db-customer --service-objective S2 — exactly one step, on exactly that database.

Guardrails

Restrict scaling to the named database.
Define minimum and maximum capacity.
Require approval for scale-up and scale-down.
Time-limit temporary capacity — an un-reversed emergency scale-up is how a P1 becomes a budget incident.
Do not autonomously force plans, terminate sessions, modify indexes, or change schema.
Track the temporary cost impact with an Azure Cost Management alert.

Use case #4 — Azure Cosmos DB: HTTP 429 throttling

The one where the correct root cause is "we're succeeding."

What happened

A marketing campaign increased Product Catalog traffic by roughly 40%. The Cosmos DB container hit its provisioned throughput ceiling. HTTP 429s increased, and client retries amplified application latency.

How Azure Monitor detected it

High normalized RU consumption
Increased HTTP 429 responses
Elevated server-side latency
Application dependency failures
Sustained operation near the throughput limit

P2 – Cosmos DB throttling affecting Product Catalog requests

How the agent found the cause

Reviewed normalized RU consumption and throttled requests.
Identified the affected database and container.
Reviewed regional and partition behavior.
Checked for hot-partition evidence.
Reviewed application retry telemetry.
Compared current traffic with the historical baseline.
Correlated demand with the marketing campaign.
Checked recent application deployments.
Checked Azure service health.
Determined the primary cause was legitimate demand.

Agent finding:

The Product Catalog container is at its configured throughput ceiling, and 21% of requests are being throttled. Traffic increased by approximately 40% following a campaign launch. No deployment or regional platform issue correlates with the event.

Step 4 is the fork in the road. If consumption is uneven across partitions, more RU/s is money set on fire — the correct answer is an architecture change, not a scale-up. The agent has to check before it recommends.

Agent action table

#	Action	Type	Detail
1	Investigate throttling	Read	RU consumption, 429 rate, latency, requests, availability
2	Identify scope	Read	Account, database, container, operations, regions
3	Analyze demand	Read	Compares traffic and RU consumption with historical patterns
4	Check partitions	Read	Looks for uneven partition consumption where telemetry permits
5	Analyze retries	Read	Determines whether client retries are amplifying the incident
6	Correlate events	Read	Links demand to campaign traffic; excludes release/platform issues
7	Assess blast radius	Read	Affected catalog operations; unaffected containers
8	Create SNOW INC	Write (ITSM)	P2 against the Product Catalog CI
9	Update INC evidence	Write (ITSM)	RU, throttling, latency, traffic, partition evidence
10	Classify response	Decision	Demand-driven unless hot-partition evidence exists
11	Calculate throughput	Read	Smallest increase within the cost ceiling
12	Request approval	Decision	Approval for a temporary throughput increase
13	Increase throughput	Write (Azure)	Raises throughput only to the approved maximum
14	Validate recovery	Validation	429 rate and latency recover
15	Update/resolve INC	Write (ITSM)	Throughput, approval, cost implication, recovery
16	Create capacity task	Write (ITSM)	Work to return throughput to normal
17	Create SNOW PRB	Write (ITSM)	Partition or retry improvements, if required

Recovery note

Provisioned throughput was increased within the approved production limit. HTTP 429 responses declined from 21% to below 1%, and Product Catalog latency returned to baseline. The increase is temporary and will be reviewed after campaign traffic subsides.

Try it yourself

Break it: the container was created at 400 RU/s. Drive a few hundred reads per second at it and you'll be throttled within seconds.

Alert it:

az monitor metrics alert create -g $RG -n "alert-cosmos-429" \ --scopes "/subscriptions/$SUB/resourceGroups/$RG/providers/Microsoft.DocumentDB/databaseAccounts/cosmos-sre-demo" \ --condition "total TotalRequests where StatusCode == 429 > 100" \ --window-size 5m --evaluation-frequency 1m --severity 2 \ --description "P2 – Cosmos DB throttling affecting Product Catalog requests"

Ask it:

cosmos-sre-demo container 'products' is throttling. Read-only. Before recommending anything, tell me whether RU consumption is EVEN across physical partitions or concentrated. If it is concentrated, do not recommend a throughput increase — recommend an architecture problem record instead. If it is even, tell me the smallest RU/s that clears throttling and the daily cost delta.

That prompt is the whole use case. Getting the agent to refuse the easy answer under a stated condition is the skill.

Verify it:

AzureDiagnostics | where ResourceProvider == "MICROSOFT.DOCUMENTDB" | where Category == "DataPlaneRequests" | summarize throttled = countif(statusCode_s == "429"), total = count() by bin(TimeGenerated, 1m) | extend throttleRate = 100.0 * throttled / total | render timechart

Guardrails

Define a maximum throughput ceiling the agent may not exceed.
Restrict changes to the named container.
Require approval for increases and reductions.
Do not permit deletion, consistency-level changes, or region changes.
Create cost alerts for prolonged increased throughput.
Treat persistent hot partitions as an architecture issue, never as a scaling issue.

Use case #5 — Azure VM: OS/root disk full

The most operationally dangerous use case in this post, and the one with the most interesting guardrail design.

What happened

A legacy payment application generated excessive trace logs. Log rotation stopped working and the root filesystem filled. The VM stayed available at the Azure platform layer — heartbeat green, Resource Health fine — but the application stopped, because it could no longer write to disk.

This is the classic "green dashboard, dead service" failure. Platform-layer monitoring alone will never catch it.

How Azure Monitor detected it

Azure Monitor Agent and guest telemetry detected:

Critically low filesystem free space
Rapid filesystem consumption
Application process stopped
Failed availability tests
Elevated HTTP 5xx
Healthy VM heartbeat but unhealthy application

P1 – Legacy payment application unavailable due to full VM OS disk

How the agent found the cause

Confirmed the VM was online.
Confirmed Azure Monitor Agent heartbeat.
Identified the affected root/OS filesystem.
Reviewed the free-space trend.
Correlated application failure with disk exhaustion.
Found 86 GB of growth in the application trace directory.
Reviewed recent deployments and logging changes.
Checked log-rotation status.
Confirmed the files matched the approved cleanup policy.
Excluded customer data, database files, audit logs, and system files.

Agent finding:

The VM is healthy at the Azure platform layer, but the OS volume has less than 1% free space. The approved application trace directory grew by 86 GB in six hours. The payment service stopped when it could no longer write to disk.

Agent action table

#	Action	Type	Detail
1	Investigate VM	Read	VM, agent heartbeat, Azure platform health
2	Analyze filesystem	Read	Root volume and free-space trend
3	Correlate app failure	Read	Service stopped after disk exhaustion
4	Identify disk consumer	Read	Abnormal growth in an approved trace directory
5	Check changes	Read	Deployments, logging changes, rotation, scheduled tasks
6	Validate cleanup scope	Read	Candidate files meet approved path, type, and age rules
7	Protect data	Decision	Excludes databases, customer data, security logs, unknown files
8	Create SNOW INC	Write (ITSM)	P1 against the payment VM/application CI
9	Update INC evidence	Write (ITSM)	Disk usage, growth timeline, service impact, cleanup scope
10	Request approval	Decision	Approval for the restricted recovery runbook
11	Archive logs	Write (Guest OS)	Archives eligible files to protected Azure Storage
12	Remove eligible files	Write (Guest OS)	Removes only successfully archived, allowlisted files
13	Run log rotation	Write (Guest OS)	Executes the approved rotation operation
14	Restart service	Write (Guest OS)	Restarts only the named payment service if necessary
15	Validate recovery	Validation	Disk, application, archive, availability, growth stabilization
16	Update/resolve INC	Write (ITSM)	Bytes processed, exclusions, approval, results
17	Create SNOW PRB	Write (ITSM)	Permanent logging and rotation remediation

Recovery note

The approved recovery runbook archived and removed 82 GB of eligible application trace files. OS-volume free space is now 31%. The payment service was restarted and has passed health checks for 15 minutes. No database, customer, security, or system files were modified.

That last sentence is not decoration. It is the sentence your auditor will read.

Try it yourself

Break it (lab VM only — this fills the root disk):

az vm run-command invoke -g $RG -n vm-payments-linux \ --command-id RunShellScript --scripts " mkdir -p /var/log/payments/trace fallocate -l 24G /var/log/payments/trace/trace-$(date +%s).log df -h / "

Ask it:

vm-payments-linux root filesystem is nearly full and the payment service is down. Read-only first. Tell me: - exactly which directory grew, by how much, over what window - whether log rotation is configured and when it last ran - what changed in the last 24 hours that would explain it Then tell me which files are inside the approved cleanup allowlist (/var/log/payments/trace/*.log, older than 2h) and which are NOT, and confirm no database, audit, or customer data files are in scope. Do not delete anything.

The critical design point. SRE Agent blocks delete and remove commands outright. You cannot have it rm those files through its Azure CLI surface, and you should be glad. The correct implementation is a fixed-purpose, version-controlled runbook that the agent invokes with tightly bounded parameters:

# What the agent is allowed to call — one runbook, allowlisted parameters, nothing else az automation runbook start \ -g $RG --automation-account-name aa-sre-runbooks \ -n "Reclaim-TraceDiskSpace" \ --parameters vmName=vm-payments-linux \ allowedPath=/var/log/payments/trace \ pattern='*.log' \ minAgeHours=2 \ maxBytes=90000000000 \ archiveToContainer=payments-trace-archive \ requireArchiveBeforeDelete=true \ dryRun=false

The runbook — not the agent — owns the destructive logic, and it enforces:

path allowlist (refuse anything outside allowedPath)
filename pattern allowlist
minimum file age
maximum total bytes per execution
successful archive verified before any deletion
hard-refuse if any candidate file is unknown, or matches a protected pattern (*.mdf, *.bak, /var/log/audit/*, *.key, *.pem)
a cooldown that prevents re-execution within N hours

Verify it:

az vm run-command invoke -g $RG -n vm-payments-linux \ --command-id RunShellScript \ --scripts "df -h /; systemctl is-active payments.service; ls -la /var/log/payments/trace | head"

Guardrails

Do not give SRE Agent unrestricted SSH or shell access. Ever. This is the single highest-leverage rule in this post.
Use a version-controlled, fixed-purpose runbook.
Allowlist paths, patterns, file ages, and maximum cleanup size.
Stop if the responsible files are unknown. A disk filled by something you can't identify is a security event until proven otherwise.
Require successful archival before deletion.
Prevent repeated execution with a cooldown.
Treat cleanup as temporary mitigation — the problem record is the fix.

Use case #6 — Linux VM: anomalous CPU saturation

Where "anomalous" is doing all the work.

What happened

Release 7.3.1 introduced an immediate retry loop when an inventory dependency failed. The application retried continuously without backoff, consuming nearly all VM CPU and causing request timeouts.

How Azure Monitor detected it

The alert deliberately combined conditions rather than firing on a threshold:

CPU significantly outside the historical baseline
Persistent saturation over multiple evaluations
Increased request latency
Increased dependency failures
No approved maintenance or batch workload active

P1 – Production order-processing Linux VM experiencing anomalous CPU saturation

A static "CPU > 90%" rule on a batch-processing VM is a pager that everyone learns to ignore. The composite condition is what makes the alert worth waking someone — or an agent — for.

How the agent found the cause

Confirmed the VM was online.
Compared current CPU with historical behavior.
Identified the order-processing service as the main CPU consumer.
Reviewed application request latency.
Reviewed downstream dependency failures.
Analyzed application retry logs.
Correlated CPU growth with release 7.3.1.
Excluded expected batch jobs.
Excluded Azure maintenance or platform issues.
Confirmed other application instances had sufficient capacity.

Agent finding:

CPU increased from a normal range of 35–50% to 98% four minutes after deployment 7.3.1. The order-processing service is repeatedly calling a failed dependency without backoff. No expected batch job or platform maintenance is active.

Agent action table

#	Action	Type	Detail
1	Confirm anomaly	Read	Compares current CPU with the historical baseline
2	Analyze impact	Read	Latency, timeouts, availability, dependencies
3	Identify process	Read	Named order service is the primary CPU consumer
4	Correlate deployment	Read	Release 7.3.1, four minutes before saturation
5	Analyze logs	Read	Dependency retry loop without backoff
6	Rule out expected work	Read	Excludes batch, backup, maintenance, scheduled processing
7	Assess blast radius	Read	Traffic and available capacity on other instances
8	Create SNOW INC	Write (ITSM)	P1 against the Order Processing CI
9	Update INC evidence	Write (ITSM)	CPU, process, dependency, release, customer impact
10	Classify response	Decision	Rollback, not VM resize or arbitrary process termination
11	Request approval	Decision	Approval to drain, roll back, restart the service
12	Drain VM	Write (Azure)	Removes the VM from load-balancer rotation
13	Restore prior release	Write (Guest OS/DevOps)	Restores known-good version or configuration
14	Restart named service	Write (Guest OS)	Restarts only `orders-service`
15	Validate recovery	Validation	CPU, retries, latency, dependencies, health recover
16	Return to rotation	Write (Azure)	Restores traffic only after successful validation
17	Update/resolve INC	Write (ITSM)	Drain, rollback, restart, approval, results
18	Create defect	Write (ITSM)	Retry, backoff, and circuit-breaker remediation

Steps 12 and 16 are the pattern to steal: drain before you touch, restore traffic only after validation passes. Most homegrown automation restarts a service while it's still taking traffic and turns a degradation into an outage.

Recovery note

Release 7.3.1 introduced a retry loop when the inventory dependency failed. After approval, the VM was drained, the previous release was restored, and orders-service was restarted. CPU declined from 98% to 43%, and application latency returned to baseline.

Try it yourself

Break it:

az vm run-command invoke -g $RG -n vm-payments-linux \ --command-id RunShellScript --scripts " nohup bash -c 'while true; do curl -s -m 1 http://127.0.0.1:9/inventory >/dev/null 2>&1; done' & nohup bash -c 'while true; do :; done' & echo started "

Ask it:

CPU on vm-payments-linux is at 98%. Read-only. First: is this actually anomalous, or is it consistent with this VM's historical pattern for this hour and day of week? Show me the baseline. If anomalous: which process, which dependency is it calling, at what rate, and what deployed immediately before? Do NOT propose killing the top process or resizing the VM.

Expect it to propose a drain → restore → restart sequence, with the drain step as a separate approval from the restart.

Guardrails

Do not automatically terminate the highest-CPU process. It is very often a legitimate workload, and occasionally it's a security incident you just destroyed the evidence for.
Allow operations only for named services.
Drain before service restart where possible.
Escalate unknown or suspicious processes to security rather than remediating them.
Limit restart attempts.
Do not permanently resize the VM when the evidence points to faulty code. Resizing to survive a retry storm is buying hardware to host a bug.

Use case #7 — Windows VM with IIS: memory leak

What happened

Release ClaimsPortal 5.9.0 introduced a memory leak in the Claims IIS application pool. Memory consumption climbed over several hours, paging began, request queues grew, and IIS returned HTTP 503.

How Azure Monitor detected it

Azure Monitor Agent collected available memory, committed bytes, paging activity, process working set, IIS request queues, HTTP 500/503 responses, and availability-test results. The alert required sustained abnormal growth, not a brief spike.

P1 – Memory exhaustion affecting production IIS application

How the agent found the cause

Confirmed the VM and monitoring agent were healthy.
Compared memory behavior with the historical baseline.
Identified the relevant w3wp.exe process.
Mapped it to the Claims application pool.
Reviewed paging and request queues.
Correlated HTTP 503 errors with low available memory.
Reviewed Windows Event Logs.
Correlated the growth with release 5.9.0.
Excluded antivirus and scheduled-reporting activity.
Confirmed another instance could carry traffic.

Agent finding:

Available memory declined from 42% to 4% over three hours. The Claims application pool grew from 1.8 GB to 11.6 GB without releasing memory after traffic normalized. Paging and HTTP 503 errors followed. The pattern began after release 5.9.0.

"Without releasing memory after traffic normalized" is the sentence that distinguishes a leak from load. Cache growth under load is normal; failure to return afterwards is not.

Agent action table

#	Action	Type	Detail
1	Confirm anomaly	Read	Memory, committed bytes, paging vs. baseline
2	Identify process	Read	Maps growing `w3wp.exe` to the Claims pool
3	Analyze IIS health	Read	Pools, queues, HTTP errors, availability
4	Correlate release	Read	Links sustained growth to release 5.9.0
5	Rule out other causes	Read	Excludes scheduled jobs, antivirus, maintenance
6	Assess capacity	Read	Confirms another instance can serve traffic
7	Create SNOW INC	Write (ITSM)	P1 against the Claims Portal CI
8	Update INC evidence	Write (ITSM)	Memory, paging, pool, release, HTTP impact
9	Classify response	Decision	Targeted recycling, not a full VM restart
10	Request approval	Decision	Approval to drain and recycle the named pool
11	Drain VM	Write (Azure)	Removes VM from load-balancer rotation
12	Capture diagnostics	Write (Guest OS)	Captures approved diagnostics to a protected location
13	Recycle app pool	Write (Guest OS)	Recycles only the Claims application pool
14	Validate recovery	Validation	Memory, paging, queues, HTTP errors, availability recover
15	Return to rotation	Write (Azure)	Restores traffic after health checks pass
16	Update/resolve INC	Write (ITSM)	Memory before/after, approval, stability
17	Create SNOW PRB	Write (ITSM)	Memory-leak remediation

Step 12 before step 13 matters: recycling the pool destroys the evidence. Capture first.

Recovery note

Abnormal memory growth was isolated to the Claims application pool following release 5.9.0. The VM was drained, the approved application pool was recycled, and health checks passed before traffic was restored. Available memory increased from 4% to 61%, and HTTP 503 responses stopped.

Try it yourself

Break it (lab VM only):

az vm run-command invoke -g $RG -n vm-claims-win ` --command-id RunPowerShellScript --scripts " Install-WindowsFeature Web-Server -IncludeManagementTools New-WebAppPool -Name 'ClaimsPool' # Simulate the leak \$leak = New-Object System.Collections.ArrayList 1..40 | ForEach-Object { [void]\$leak.Add((New-Object byte[] 100MB)) ; Start-Sleep -Milliseconds 200 } "

Ask it:

vm-claims-win available memory is at 4%. Read-only. Map the growing process to an IIS application pool. Show me the memory curve for the last 6 hours and tell me whether memory was released after traffic dropped. Correlate with deployment history. Then propose the most targeted possible mitigation — I do not want a VM restart.

Expect it to propose:

Restart-WebAppPool -Name "ClaimsPool"

…and nothing else on that machine.

Guardrails

Recycle only the named application pool.
Avoid full VM restart as the first response — it's a bigger hammer with a longer outage and it destroys the leak evidence.
Do not copy memory dumps into ServiceNow. They contain credentials, tokens, and customer data.
Store diagnostics in a secured location; put the link in the ticket.
Prevent repeated automatic recycling — a pool that needs recycling every 40 minutes is an incident, not a routine.
Escalate if the leak returns during the observation period.

Use case #8 — Virtual Machine Scale Set: unhealthy instance

The most autonomy-ready use case in the list, and the reason is stateless workloads.

What happened

A configuration extension failed while VMSS instance 17 was being provisioned. The VM was running but its application service never started. The instance failed application health probes and caused intermittent errors.

How Azure Monitor detected it

Reduced healthy backend count
Application Health extension failure
Backend health-probe failure
VM extension provisioning failure
Instance-specific errors
Elevated intermittent HTTP 5xx

P2 – Unhealthy VM Scale Set instance causing intermittent API failures

How the agent found the cause

Reviewed the VMSS healthy-instance count.
Identified instance 17 as the only unhealthy instance.
Reviewed backend health.
Compared instance 17 with healthy instances.
Checked the image and VMSS model.
Reviewed VM extension state.
Found the configuration extension failure.
Reviewed boot and application diagnostics.
Confirmed the workload was stateless.
Confirmed eleven instances could carry production traffic.

Agent finding:

Instance 17 is the only unhealthy member of the 12-instance scale set. Its application health probe has failed since 09:18 UTC. The configuration extension failed during provisioning, and the application service never started. Eleven healthy instances can maintain service.

Agent action table

#	Action	Type	Detail
1	Investigate VMSS	Read	Instance health, provisioning state, healthy count
2	Identify instance	Read	Instance 17 is the only unhealthy member
3	Analyze backend health	Read	Confirms the instance fails application probes
4	Compare instances	Read	Image, model, extensions, configuration
5	Find extension failure	Read	Locates the failed configuration extension
6	Review diagnostics	Read	Boot, extension, and application diagnostics
7	Assess safe capacity	Read	Eleven instances can carry traffic
8	Confirm statelessness	Decision	Replacement won't destroy required local state
9	Create SNOW INC	Write (ITSM)	P2 against the VMSS/application CI
10	Update INC evidence	Write (ITSM)	Instance, extension error, health, capacity evidence
11	Classify response	Decision	Replacement or reimage per approved procedure
12	Request approval	Decision	Approval to isolate and replace instance 17
13	Isolate instance	Write (Azure)	Ensures the instance receives no production traffic
14	Preserve evidence	Read/Write	Stores approved diagnostic evidence securely
15	Replace instance	Write (Azure)	Reimages or replaces only instance 17
16	Validate provisioning	Validation	Image, model, and extensions deploy successfully
17	Validate service	Validation	Backend health, capacity, customer errors recover
18	Update/resolve INC	Write (ITSM)	Replacement, approval, diagnostics, recovery
19	Create SNOW PRB	Write (ITSM)	Extension and image-validation improvement work

Steps 8 and 7 are the gate. Replacing a stateless instance with eleven healthy peers is genuinely low risk. Replacing a stateful instance, or replacing one when you're already at minimum capacity, is an outage. Both must be confirmed, not assumed.

Recovery note

VMSS instance 17 was isolated after its configuration extension failed and the application service did not start. Diagnostic evidence was captured, and the instance was replaced after approval. The replacement passed extension, application, and backend health checks. The scale set has returned to 12 healthy instances.

Try it yourself

Break it:

INSTANCE_ID=$(az vmss list-instances -g $RG -n vmss-api-demo \ --query "[5].instanceId" -o tsv) az vmss extension set -g $RG --vmss-name vmss-api-demo \ --name CustomScript --publisher Microsoft.Azure.Extensions \ --settings '{"commandToExecute":"exit 1"}'

Ask it:

vmss-api-demo has an unhealthy instance. Read-only. Identify which instance, since when, and the specific extension error. Confirm for me: (a) the workload is stateless, (b) how many healthy instances remain, and (c) whether remaining capacity can carry current traffic with 20% headroom. Only if all three are satisfied, propose a reimage of that single instance. Capture diagnostics before proposing anything.

Expect it to propose:

az vmss reimage -g $RG -n vmss-api-demo --instance-id $INSTANCE_ID

Verify it:

az vmss get-instance-view -g $RG -n vmss-api-demo --instance-id $INSTANCE_ID \ --query "vmHealth.status.code" az vmss list-instances -g $RG -n vmss-api-demo -o table

Guardrails

Confirm the workload is stateless before any replacement.
Confirm sufficient healthy capacity first.
Preserve diagnostic evidence before replacement — the instance is your only copy of the failure.
Restrict permissions to the named VMSS.
Limit simultaneous replacements. One at a time. An agent that reimages six instances because six probes failed has just caused the outage it was investigating.
Do not permit deletion of the entire scale set.

Use case #9 — Application Gateway: HTTP 502 from unhealthy backends

The cross-component change nobody coordinated.

What happened

Customer Portal release 9.2 changed the backend listener from port 443 to 8443. Application Gateway remained configured to connect on 443. All backend probes failed and customers received HTTP 502.

How Azure Monitor detected it

Increased Application Gateway HTTP 502 responses
Increased failed requests
Four unhealthy backends
Reduced healthy-host count
Failed synthetic availability tests

P1 – Application Gateway returning HTTP 502 due to unhealthy backends

How the agent found the cause

Reviewed Application Gateway metrics.
Retrieved backend health.
Identified the affected pool.
Reviewed probe path, protocol, host header, and port.
Reviewed backend settings.
Confirmed the application responded on 8443.
Confirmed the gateway used 443.
Correlated the mismatch with release 9.2.
Reviewed NSG, route, DNS, certificate, and WAF changes.
Excluded networking, certificate, and platform-health issues.

Agent finding:

HTTP 502 responses began at 16:07 UTC. All four Customer Portal backends are unhealthy. Release 9.2 changed the backend listener to port 8443, while Application Gateway continues to use port 443. No NSG, routing, or certificate issue correlates with the incident.

Step 9 is what separates a real investigation from a lucky guess. A 502 has at least six plausible causes — NSG, UDR, DNS, expired cert, WAF rule, backend down. The agent has to exclude them, in writing, before you trust the conclusion.

Agent action table

#	Action	Type	Detail
1	Investigate gateway	Read	HTTP 502, failed requests, latency, backend counts
2	Inspect backend health	Read	Identifies four unhealthy Customer Portal backends
3	Review configuration	Read	Settings, probes, protocol, port, TLS, routing
4	Test backend state	Read	App responds on 8443 while gateway uses 443
5	Correlate changes	Read	Links mismatch to Customer Portal release 9.2
6	Rule out networking	Read	NSGs, routes, DNS, TLS, WAF, platform health
7	Assess blast radius	Read	Confirms the Customer Portal pool is unavailable
8	Create SNOW INC	Write (ITSM)	P1 against the portal/gateway CI
9	Update INC evidence	Write (ITSM)	502s, backend, port, deployment, known-good configuration
10	Classify response	Decision	Restore the known-good backend listener
11	Request approval	Decision	Approval to restore source-controlled configuration
12	Restore listener	Write (Application)	Restores the application listener to approved port 443
13	Validate backend health	Validation	All four backends become healthy
14	Validate application	Validation	HTTP 502 declines; synthetic transactions pass
15	Verify controls	Read	Confirms no WAF, TLS, routing, or NSG control was weakened
16	Update/resolve INC	Write (ITSM)	Cause, restoration, approval, recovery
17	Create SNOW CHG	Write (ITSM)	Coordinated change for the intended port migration
18	Create SNOW PRB	Write (ITSM)	Cross-component deployment-validation work

Step 12 is a genuinely important choice. There were two ways to fix this: change the application back to 443, or change the gateway to 8443. The agent restores the application to the known-good, source-controlled state rather than mutating the gateway to match an unapproved change. One of those is a rollback; the other is ratifying an unreviewed change during an outage. Then step 17 files a proper change record for the migration the team clearly intended to do.

Step 15 exists because the fastest way to make a 502 disappear is to disable TLS validation. The agent must prove it didn't take the fast way.

Recovery note

Customer Portal release 9.2 changed the backend listener from port 443 to 8443 without a coordinated gateway change. The application listener was restored to the previous configuration. All four backends are healthy, HTTP 502 responses returned to baseline, and synthetic login tests passed for 15 minutes.

Try it yourself

Ask it:

appgw-portal is returning 502s and all backends are unhealthy. Read-only. Walk me through the exclusion, explicitly, for each of: NSG, UDR/route table, DNS resolution, backend TLS certificate, WAF rule blocking, backend process down, and probe configuration mismatch. State which you ruled out and the evidence for each. Then tell me the known-good configuration and where it is source-controlled.

Verify it:

az network application-gateway show-backend-health \ -g $RG -n appgw-portal \ --query "backendAddressPools[].backendHttpSettingsCollection[].servers[].{addr:address,health:health}" -o table

Guardrails

Do not disable TLS validation. Not to test, not temporarily, not "just to confirm."
Do not weaken NSGs or WAF policies as a mitigation.
Use source-controlled configuration as the definition of "known-good."
Restore a known-good state during the incident; migrate through a change record afterwards.
Require approval for gateway or backend changes.
Validate full application transactions, not just health probes. A probe returning 200 on /health proves very little.

Use case #10 — Azure Service Bus: queue and dead-letter backlog

The one that teaches the most important lesson in the entire post.

What happened

Release fulfillment-worker:6.4.0 couldn't deserialize messages containing a new deliveryWindow field. Consumer throughput dropped by 92%. The active backlog grew rapidly, and incompatible messages entered the dead-letter queue.

How Azure Monitor detected it

Increasing active-message count
Increasing oldest-message age
Dead-letter growth
Reduced completed-message rate
Consumer application errors
Delayed downstream business processing

P1 – Production order-event backlog delaying fulfillment

How the agent found the cause

Reviewed active, incoming, outgoing, and dead-letter counts.
Calculated message arrival and completion rates.
Confirmed the backlog was growing.
Reviewed consumer instance health.
Reviewed consumer errors and restarts.
Checked downstream dependency health.
Checked Service Bus authentication and authorization.
Correlated the throughput decline with release 6.4.0.
Found deserialization errors for the new field.
Determined scaling more broken consumers would amplify failures.

Agent finding:

The order-events queue grew from 3,000 to 185,000 active messages in 35 minutes. Consumer throughput dropped by 92% immediately after release fulfillment-worker:6.4.0. Application logs show deserialization failures involving the new deliveryWindow field.

Step 10 is the whole reason to use a reasoning agent instead of an autoscale rule. Every metric here screams "scale out the consumers." An HPA would have done exactly that, and every new replica would have dead-lettered messages faster.

Agent action table

#	Action	Type	Detail
1	Investigate queue	Read	Active, incoming, outgoing, scheduled, DLQ counts
2	Calculate flow rates	Read	Confirms arrival exceeds completion; backlog growing
3	Inspect consumers	Read	Health, instance count, scaling, errors, dependencies
4	Analyze failures	Read	Deserialization exceptions involving the new field
5	Correlate release	Read	Links the 92% throughput reduction to release 6.4.0
6	Rule out Service Bus	Read	Health, authorization, throttling, networking, service status
7	Assess blast radius	Read	Backlog age and fulfillment impact
8	Protect message data	Decision	Prevents payloads or personal data entering ServiceNow
9	Create SNOW INC	Write (ITSM)	P1 against the fulfillment integration CI
10	Update INC evidence	Write (ITSM)	Backlog, age, flow, exception, release, business impact
11	Classify response	Decision	Rollback, not scaling broken consumers
12	Request approval	Decision	Approval to restore consumer 6.3.7
13	Roll back consumer	Write (Azure/K8s)	Rolls back through the approved deployment platform
14	Restore capacity	Write (Azure/K8s)	Restores approved consumer instance count
15	Validate processing	Validation	Consumer and downstream health
16	Validate backlog	Validation	Completion exceeds arrival; DLQ growth stops
17	Update INC	Write (ITSM)	Rollback, rates, estimated drain time, approval
18	Maintain incident	Decision	Keeps the incident open until backlog age meets the objective
19	Create SNOW CHG	Write (ITSM)	Separate controlled change for DLQ replay
20	Create SNOW PRB	Write (ITSM)	Message-contract compatibility remediation

Recovery note

Consumer release 6.4.0 could not deserialize messages containing the new deliveryWindow field. The fulfillment worker was rolled back to 6.3.7. Consumer throughput recovered, new dead-letter growth stopped, and the active backlog is draining at approximately 7,500 messages per minute. Dead-letter replay requires a separately approved procedure.

The lesson: technical recovery is not business recovery

Step 18 is the most important row in this entire post.

At the moment of rollback, every technical signal is green. Consumers are healthy. Throughput has recovered. The DLQ has stopped growing. An agent optimizing for metrics would resolve the incident right there and go back to sleep.

But there are still 185,000 unshipped orders and a dead-letter queue full of messages that need a separately approved replay procedure. Customers are still affected. The incident stays open until backlog age meets the business objective, not until the graphs look nice.

Encode this in the response plan explicitly:

Do not resolve this incident when consumer health recovers. Resolution criteria: 1. Completion rate exceeds arrival rate for 15 consecutive minutes, AND 2. Oldest active message age is under 5 minutes, AND 3. Dead-letter count has not increased for 30 minutes. DLQ replay is out of scope for this incident. File a separate change record.

Try it yourself

Break it:

# Flood the queue while no consumer is running for i in $(seq 1 5000); do az servicebus queue message send -g $RG --namespace-name sb-sre-demo \ -q order-events --body "{\"orderId\":$i,\"deliveryWindow\":\"2026-08-09T10:00Z\"}" 2>/dev/null done az servicebus queue show -g $RG --namespace-name sb-sre-demo -n order-events \ --query "countDetails" -o json

Ask it:

order-events on sb-sre-demo has a growing backlog. Read-only. Give me: arrival rate, completion rate, current active count, oldest message age, DLQ count and DLQ growth rate, and the projected drain time at current rates. Then tell me why scaling out consumers is or is not the correct action here. Do not include any message payloads or customer data in your answer.

That last line is not optional. Message bodies routinely contain names, addresses, and payment references — and everything the agent writes goes into a ticket.

Verify it:

az servicebus queue show -g $RG --namespace-name sb-sre-demo -n order-events \ --query "{active:countDetails.activeMessageCount, dlq:countDetails.deadLetterMessageCount}" -o json

Guardrails

Never automatically purge queues.
Do not automatically replay dead-letter messages. Replay without idempotency guarantees means duplicate charges and duplicate shipments.
Do not place message payloads in ServiceNow.
Confirm idempotency before replay.
Scale consumers only when the processing path is healthy.
Require separate approval for replay or queue configuration changes.
Keep the incident open until business recovery is confirmed.

7. The ITSM integration model

Recommended incident fields

Field	Example
Short description	Production Orders API pods failing after deployment
Configuration item	`prod-aks-orders-api`
Assignment group	Container Platform Operations
Impact	High
Urgency	High
Environment	Production
Azure resource ID	Full affected Azure resource ID
Azure alert ID	Azure Monitor alert correlation identifier
SRE investigation	Link to the SRE Agent investigation thread
Current impact	Eight of ten replicas unavailable
Probable cause	Missing configuration in latest release
Confidence	High
Proposed action	Roll back to revision 41
Approval	Approver and UTC timestamp
Validation	Ten replicas ready and synthetic tests passing
Resolution	Service restored through rollback

The Confidence field earns its place. An agent that says "probable cause: missing configuration (confidence: low)" is far more useful than one that always sounds certain, because it tells the human how much to verify before approving.

Fields the agent can set directly (preview): assignment_group, category, subcategory, impact, urgency, priority, short_description, and any custom u_* field. It cannot change incident state through field updates — acknowledge and resolve are separate, dedicated tools.

Recommended agent-generated timeline

Every work note should carry:

Element	Example
Timestamp	`2026-03-18 14:26 UTC`
Observation	HTTP 500 rate increased to 18%
Evidence	Link to the Application Insights query
Change correlation	Incident started three minutes after deployment
Probable cause	Missing production application setting
Confidence	High
Proposed action	Swap to previous healthy slot
Risk	Temporary deployment rollback
Approval	Incident commander and timestamp
Execution	Slot-swap operation and result
Validation	5xx below 1% for 15 minutes
Follow-up	Problem record for configuration validation

Note that Evidence is a link, not a paste. This is a deliberate data-protection pattern: the ticket carries a pointer to the query, and the query results stay in the system that already has the right access controls.

Deduplication strategy

Do not create one incident per alert, pod, queue, or VMSS instance. Correlate on:

application/business service + Azure resource ID + environment (production) + alert-rule family + active incident time window

Related alerts attach to the existing incident as evidence or child alerts. Azure Monitor already merges recurring alerts into a single thread when it's the bound platform; for ServiceNow, this correlation key is yours to implement.

Get this wrong and your first AKS incident produces eight incidents, eight investigations, and eight rollback proposals for the same deployment.

Record responsibilities

Record	Purpose	Example
Incident (INC)	Restore service quickly and safely	App Service HTTP 500 outage
Problem (PRB)	Identify and remove the underlying cause	Missing deployment configuration validation
Change (CHG)	Govern permanent or higher-risk production changes	Coordinated Application Gateway port migration
Engineering defect/task	Correct application or automation behavior	Add retry backoff to the Linux application

The agent must clearly distinguish temporary mitigation from permanent correction. Every single use case above ends with a follow-up record, and that's not bureaucratic theatre — an agent that mitigates flawlessly and never files a problem record is an agent that lets the same outage recur forever while making the metrics look great.

8. Reality check: where you have to build

This is the section that will save you a month.

The PDF this post is built from is explicit that these are target response patterns, not guaranteed zero-configuration behavior. Having now checked each pattern against the product documentation, here is exactly where the gaps are and what fills them.

#	The pattern assumes	What's actually documented	What you must build
1	Agent creates a ServiceNow INC	ServiceNow is an inbound platform. Documented writes: post discussion entries, acknowledge, resolve, plus field updates (preview)	Incidents should originate in ServiceNow (via its own Azure Monitor integration) and flow in. If you truly need agent-initiated creation, add a Python tool or MCP server against the ServiceNow Table API
2	Agent creates PRB / CHG / defect records	Not a documented first-class ServiceNow action	Same: a custom tool against `/api/now/table/problem` and `/change_request`. This is ~30 lines of Python and worth doing properly, with a least-privileged integration user
3	ServiceNow and PagerDuty both connected	Only one incident platform active at a time; switching disconnects the other	Bind the agent to your system of record (ServiceNow). Reach the pager through a connector, Teams/Slack, or a webhook
4	Agent runs guest-OS cleanup (`rm`, rotate, restart)	`delete` and `remove` commands are blocked outright. `az keyvault` blocked. Management locks respected	Wrap all guest-OS work in fixed-purpose Azure Automation runbooks or a constrained `az vm run-command` script, invoked with allowlisted parameters. See use case #5
5	Agent "pauses the release pipeline"	Requires the GitHub or Azure DevOps connector, plus permissions on that pipeline	Connect source control; grant pipeline permissions explicitly; test the pause path before you need it
6	Approval gate on every action	Review mode shows Approve/Deny only for Azure infrastructure operations. Emails, Teams posts, and external queries proceed on the agent's reasoning	Use hooks or tool access policies to gate non-Azure actions
7	Agent resolves incidents	Supported — but during a pilot you don't want it	Require human confirmation before resolve. Encode it in the response plan and enforce it with a `Stop` hook
8	One agent handles everything	Response plans route to custom agents; skills cap at five concurrent active	Build domain specialists (§4.6). A single mega-agent thrashes its skill budget
9	Autonomous mode by default is fine	Per-plan default is Autonomous, and connecting a platform auto-creates an autonomous `quickstart_handler`	Delete the quickstart plan. Set every plan to Review explicitly
10	Agent has broad subscription rights	You can't remove individual permissions — only whole resource groups	Design resource groups as blast-radius boundaries before onboarding

None of these are blockers. All of them are a week of work you'd rather discover now than during your pilot readout.

9. Approval and autonomy policy

The policy I would actually ship:

Action category	Recommended initial policy
Read metrics, logs, traces, resource health	Automatic
Correlate deployments and configuration changes	Automatic
Create a ServiceNow incident	Automatic after deduplication
Add sanitized work notes	Automatic
Prepare a remediation plan	Automatic
Create a draft ServiceNow change	Automatic, but not approve it
Modify an Azure production resource	Human approval required
Execute a VM guest runbook	Human approval required
Roll back an application	Human approval required
Delete or replace a stateless VMSS instance	Human approval required
Resolve a ServiceNow incident	Human confirmation during the pilot
Delete data, purge queues, replay DLQ messages	Separate explicit approval
Autonomous remediation	Only after a proven, bounded pilot

Two notes on making this real:

Start in Review and stay there longer than feels necessary. The documented recommendation is to observe for two to four weeks and then promote specific triggers you consistently approve. Not the agent — the triggers. Promotion should be per-response-plan and evidence-based: "we approved this exact rollback proposal eleven times without modification" is a reason to go autonomous. "It seems good" is not.

Autonomy should be earned per action type, not per environment. "Autonomous in staging" is a fine starting rule, but the durable version is "autonomous for VMSS single-instance reimage where the workload is stateless and healthy capacity exceeds 80%" — a narrow, well-characterized action with a mechanical precondition.

10. Cross-cutting security controls

Use a dedicated managed identity for SRE Agent.
Scope Azure roles to selected resources or resource groups.
Avoid broad Contributor and Owner assignments.
Use fixed-purpose Automation runbooks for guest operations.
Do not provide unrestricted SSH, shell, or PowerShell execution.
Use a dedicated least-privileged ServiceNow integration identity.
Prefer OAuth or a managed connector over stored credentials.
Store required secrets in Key Vault — never in prompts. (The agent blocks az keyvault commands entirely, which helps.)
Sanitize logs before posting them into ServiceNow.
Do not post tokens, personal data, SQL text, message payloads, or memory dumps.
Record every approval and production action.
Define remediation cooldowns and maximum retry counts.
Require post-action application validation.
Retain existing manual runbooks as fallback.
Use Azure Cost Management alerts for temporary scaling actions.

What the platform gives you for free

Worth knowing so you don't rebuild it:

Layer	Isolation model
Compute	Dedicated sandbox (micro VM) per agent; tool execution separate from the reasoning loop
Database	Separate database per agent
Blob storage	Separate blob storage per agent
Network	Per-agent proxy instance validating every outbound request
Credentials	Identity sidecar issues short-lived, per-call tokens; credentials never enter the reasoning context

Token lifetimes: managed identity ~1 hour (auto-refreshed), OAuth refreshed 20 minutes before expiry, per-tool-call action tokens are single-use, blob SAS 1 hour refreshed at 45 minutes. Each tool invocation launches a fresh process whose entire tree terminates on completion — there are no persistent process pools, so one tool call cannot see another's environment.

The audit trail

Every az command is logged to your Application Insights as an AgentAzCliExecution custom event. This is your evidence for change management:

customEvents | where name == "AgentAzCliExecution" | where timestamp > ago(30d) | project timestamp, command = tostring(customDimensions.command), resource = tostring(customDimensions.resourceId), succeeded = tostring(customDimensions.success), thread = tostring(customDimensions.threadId) | order by timestamp desc

Run that query in front of your auditor once and most of the "but can we prove what it did" conversation ends.

11. A 30/60/90 pilot that survives contact with your CAB

Phase	Enabled capability
1	Detect Azure Monitor alerts
2	Create or correlate ServiceNow incidents
3	Perform read-only investigation
4	Add sanitized findings to ServiceNow
5	Recommend remediation without execution
6	Execute bounded actions after approval in Review mode
7	Validate technical and business recovery
8	Prepare incident resolution and follow-up records
9	Consider autonomy only for proven low-risk actions

Phases 1–5 have zero production write risk and deliver most of the MTTR reduction. Do not rush past them to get to the demo-friendly part.

The best first five candidates

App Service deployment-slot rollback — clean trigger, reversible action, unambiguous validation
AKS deployment rollback — same shape, one GitOps caveat
VMSS unhealthy-instance replacement — stateless, bounded, easy precondition check
Restricted VM disk-recovery runbook — high toil, high value, forces you to build the runbook pattern properly
Automatic ServiceNow incident creation and timeline updates — the compounding one; every incident from here on is better documented than any incident before it

These five share the properties you want: clear triggers, tightly bounded actions, measurable validation criteria, and practical escalation paths.

What to measure in week one

Before you enable a single write action, capture your baseline:

Median time from alert to first accurate human diagnosis
Percentage of incidents where the first hypothesis was wrong
Median time from diagnosis to mitigation
Percentage of incidents with a complete timeline in the ticket
Percentage of incidents that produced a follow-up problem record

The read-only phase moves the first, second, and fourth of those immediately. If it doesn't, your telemetry is the problem, not the agent — and that's a genuinely useful thing to discover in week one rather than week twelve.

12. Measuring whether it's actually working

Under Monitor → Incident metrics:

Metric	What it shows
Incidents reviewed	Total incidents the agent processes
Mitigated by agent	Resolved autonomously
Assisted by agent	Agent helped; a human completed it
Mitigated by user	Human resolved using agent-provided information
Pending user action	Waiting on a human

The counter-intuitive read: "Assisted by agent" and "Mitigated by user" are the healthy numbers during a pilot. A high "Mitigated by agent" count in month one means someone left autonomy on.

Watch Pending user action closely. A growing queue there means either your approval routing is broken or the agent is proposing things nobody is comfortable approving — both are important signals, and both are invisible without this dashboard.

Also check Monitor → Session insights periodically. Each insight card links back to the thread that generated it, so you can trace any learned pattern to its origin. If the agent has learned something wrong, this is where you find it — and #forget is how you fix it.

13. Resources

Core documentation

Incident response

Extensibility

Closing

The framing that makes this work isn't "AI runs my production." It's this:

Your agent is the most junior person on the rotation — and the most thorough. It will never skip the Resource Health check. It will never forget to compare against the previous slot. It will never write "restarted it, seems fine" in a work note at 4 AM. And it will never, ever be allowed to rm -rf anything.

Every guardrail in this post exists to keep it in that role. Scope the identity to resource groups. Keep production writes in Review. Wrap guest-OS work in runbooks with allowlists. Never let it purge a queue or replay a dead-letter message on its own. Keep the incident open until customers are actually served, not until the graphs look nice.

Do that, and the ten workflows above stop being a slide deck and start being your Tuesday.

Start with use case #1. One App Service, one slot, one alert rule, one response plan in Review mode. Watch it assemble an evidence chain you'd have spent twenty minutes building by hand, and then decide how much further you want to go.

The ten scenarios in this post are target response patterns. Each one requires appropriate telemetry, scoped Azure RBAC, response-plan instructions, approved remediation tooling, and ITSM integration. Confirm current Azure SRE Agent and ServiceNow connector capabilities against Microsoft documentation before implementing — the product is moving quickly, and several capabilities referenced here are in preview.

Grok 4.6 comes to Microsoft Foundry Models: Built for long-horizon reasoning and complex workflows

MaheshBalachandran — Wed, 26 Aug 2026 17:04:55 GMT

The next wave of AI applications won't be defined by how quickly models answer questions. They'll be defined by how much work they can complete. Whether resolving repository issues, navigating codebases, generating engineering designs, or executing multi-step enterprise workflows, developers are increasingly building systems that need models capable of staying on task across complex, long-running objectives.

Today, Grok 4.6 from SpaceXAI is available in Microsoft Foundry Models through public preview, bringing SpaceXAI's latest frontier model to developers through a unified platform for model discovery, evaluation, deployment, and governance.

Built on SpaceXAI's 1.5T-scale model family, Grok 4.6 focuses on strong performance across coding, engineering, office productivity, research enablement, and inference optimization tasks. Rather than optimizing for a single benchmark category, Grok 4.6 is designed for sustained reasoning and execution across software engineering, agentic workflows, and technical problem solving.

Long-Running Agents That Stay On Task

Many AI systems perform well on isolated prompts. Real-world agents are different.

They need to plan, recover from errors, navigate tools, and continue making progress over extended workflows with minimal supervision.

Grok 4.6 is designed for this type of long-horizon agentic work, delivering strong performance on benchmarks that measure sustained software engineering and agent execution.

Benchmark highlights

Terminal-Bench 3.0

* Benchmarks are from model provider, see details.

Terminal-Bench 3.0 evaluates a model’s ability to complete multi-step, terminal-based execution that closely resemble real-world engineering workflows. For developers building coding agents, software engineering copilots, or autonomous workflows, Grok 4.6 is optimized not only for generating output, but for completing work.

Engineering-Grade Reasoning Beyond Software

Grok 4.6 extends beyond traditional coding into technical and engineering workflows. From computer-aided design (CAD) and procedural design, to complex engineering analysis, it is designed to reason through structured technical challenges and support multi-step problem solving.

Benchmark highlights

3DCodeBench

* Benchmarks are from model provider, see details.

3DCodeBench evaluates agentic procedural 3D modeling via code, testing how effectively models can reason about and generate complex engineering designs through code. The result highlights Grok 4.6's potential beyond traditional software development, supporting technical workloads across manufacturing, robotics, product design, and industrial AI.

Enterprise Agents That Deliver Work, Not Just Answers

The most valuable AI workflows often don't end with an answer. They produce business-ready deliverables, such as reports, presentations, spreadsheets, recommendations, and other business artifacts that can be acted on immediately.

Grok 4.6 demonstrates strong performance on evaluations designed to measure knowledge work and enterprise productivity.

Benchmark highlights

AA Briefcase

* Benchmarks are from model provider, see details.

AA Briefcase (Artificial Analysis) evaluates agents on long-horizon, complex professional knowledge-work projects that culminate in deliverables such as spreadsheets, presentations, memos, financial models, and PDFs. The results highlights Grok 4.6’s ability to support enterprise assistants, research agents, and business workflow automation that transform information into actionable business outcomes.

Pricing

Model	Deployment	Input/1M Tokens	Output/1M Tokens	Cache/1M Tokens
Grok 4.6	Global Standard	$2.00	$6.00	$0.50

Why build with Grok 4.6 in Foundry?

As organizations adopt more frontier models, the challenge is no longer accessing models. It's operationalizing them.

Foundry provides a unified platform to discover, evaluate, and deploy models while applying enterprise-grade governance, security, and operational controls throughout the AI lifecycle. Organizations can assess Grok 4.6 against their own workloads, business requirements, and production constraints, enabling confident model selection and faster deployment.

With Grok 4.6 in Foundry, developers can:

Evaluate Grok 4.6 alongside other leading frontier models
Run workload-specific evaluations before deployment
Deploy through managed endpoints
Integrate the model into agentic applications and workflows
Operate with enterprise governance and security controls

The combination of Grok 4.6's strengths in long-horizon execution, engineering reasoning, and enterprise productivity with Foundry's model platform capabilities gives developers another powerful option for building the next generation of AI applications.

Get started

Grok 4.6 is now available in Microsoft Foundry Models. Explore the model card to learn more and get started.

If you're building coding agents, engineering copilots, research assistants, or enterprise automation solutions, Grok 4.6 offers a compelling combination of reasoning, execution, and technical depth, all through the unified Foundry platform experience.

Azure DNS + Traffic Manager linked records

atiy — Wed, 26 Aug 2026 15:59:25 GMT

Introduction

Traffic Manager linked records creates a direct, managed link between an Azure DNS record set and a Traffic Manager profile. When a query arrives, Azure DNS evaluates the linked Traffic Manager profile internally and returns the appropriate endpoint response directly. For A and AAAA records, this means the client receives endpoint IP addresses without an intermediate CNAME response to trafficmanager.net.

This practical guide uses a small multi-region Contoso scenario to explain the architecture, configure the feature in the Azure portal, validate the DNS response, and demonstrate endpoint failover. The feature is currently in public preview, so the walkthrough should be tested in a non-production environment before broader adoption.

Note: DNS-based global routing is powerful, but the traditional integration between a custom domain and Traffic Manager adds an intermediate name that customers must understand, resolve, and govern.

The scenario and the problem

Contoso operates a public application across North America, Europe, and Asia. Users browse to contoso.com. Azure Traffic Manager monitors the regional endpoints and selects the preferred healthy destination according to the profile routing method.

Traditional CNAME integration can expose the intermediate trafficmanager.net name in the DNS response.
A CNAME cannot be placed at the zone of apex, which makes root-domain routing harder.
An unsigned intermediate DNS hop can interrupt end-to-end DNSSEC validation.
Operations teams must understand which system is authoritative for the customer zone, and which system makes the routing decision.

Design objective

Preserve Traffic Manager routing methods, endpoint health monitoring, and automatic failover while returning the effective endpoint answer directly from Azure DNS.

The solution: Traffic Manager linked records

Figure 1. Traditional CNAME resolution compared with Traffic Manager linked record.

As you see in figure 1 above, the left side shows the traditional path. The client queries the application name in Azure DNS, receives a CNAME that points to a trafficmanager.net profile name, and then performs another lookup before receiving the selected endpoint address. The right side shows the linked-record path. Azure DNS remains authoritative for the Contoso zone, consults the linked Traffic Manager profile internally, and returns the selected endpoint directly.

The key point is that the Traffic Manager still performs the routing decision and health evaluation. Linked records change the DNS integration and the answer returned to the client; they do not turn Traffic Manager into a reverse proxy or place it in the application's data path.

Reference architecture

Figure 2. Practical multi-region architecture for testing direct resolution and failover.

Reference architecture in figure 2 above shows the client queries contoso.com. Azure Public DNS hosts the authoritative zone and contains a record set linked to the tm-profile Traffic Manager profile. The profile uses a configured routing method and continuously evaluates the health of the regional endpoints. Azure DNS uses the Traffic Manager decision and returns the effective endpoint answer to the client.

For a simple lab, two endpoints are enough. Configure the first endpoint with priority 1 and the second with priority 2. The third regional endpoint in the diagram illustrates how the same design scales to additional regions. The record set can be created at the zone apex by leaving the record name empty, or for a subdomain such as www.

Note: Traffic Manager is DNS based. After name resolution, the client connects directly to the selected public endpoint. Application traffic does not pass through the Traffic Manager.

Prerequisites

An Azure subscription and permissions to create or update Azure DNS and Traffic Manager resources.
A public domain hosted in Azure DNS and delegated to the Azure DNS name servers.
Two public application endpoints that can return visibly different responses for failover testing.
The Microsoft.Network resource provider is registered in every subscription that contains the DNS zone or Traffic Manager profile.

Implementation in the Azure portal

The portal workflow has two main parts: create a strictly typed Traffic Manager profile, then create an Azure DNS record set that links to that profile.

Create the Traffic Manager profile

Step1: Open Traffic Manager profiles, select create.

Step 2: Configure the profile and instance details, fill out the subscription, resource group, name(tm-profile), routing method: priority, record type: A, and resource group location.

Step 3: Add endpoints (Add tmendpoint-1 as the public IP address and external endpoint type with priority 1.

Step 4: Repeat Step 3 to add the secondary endpoint (Add tmendpoint-2 as the public IP address and external endpoint type with priority 2)

Step 5: Confirm endpoint health (Wait until both endpoints report online before testing DNS behavior.)

Note: The record type selection creates a strictly typed profile. Select A for IPv4 answers, AAAA for IPv6 answers, or CNAME for canonical-name answers. The type is used to validate compatibility between the Traffic Manager profile and the linked DNS record.

Create the linked record in Azure DNS

Step 1: Open the Azure DNS zone, select the zone that hosts the application domain, for example contoso.com.

Step 2: Add DNS record sets, select + record set.

Step 3: Choose the record name, leave name empty for the zone apex or enter www for a subdomain.

Step 4: Choose the type, select the record type that matches the Traffic Manager profile, for example, A.

Step 5: Enable Traffic Management, select Enable Traffic Management (Preview).

Step 6: Select the profile, choose the profile subscription and tm-profile, then save the record set.

Note: The record set contains a managed association to the Traffic Manager profile. The linked record inherits its TTL from the Traffic Manager profile.

In the Azure DNS record list, the new record should appear as a linked record associated with the Traffic Manager profile. For an apex A record, the conceptual result is:

Validate the DNS response

Use both the Azure portal and direct DNS query. The portal confirms the resource association; the DNS query confirms the serving behavior.

Portal validation

The record exists in the expected Azure DNS zone.
The record type matches the strictly typed Traffic Manager profile.
The linked profile is tm-profile.
The Traffic Manager endpoints show their expected health state.

Command-line validation

A successful A-record test returns the selected healthy endpoint IP address and does not include a trafficmanager.net CNAME in the answer. Querying for an Azure DNS authoritative name server directly is useful during testing because it reduces ambiguity from local recursive resolver caches.

The specific endpoint returned depends on the routing method, endpoint health, source resolver location, and cached TTL behavior. Validate the result against the profile configuration rather than expecting one universal IP address.

Prove automatic failover

A practical demonstration should show that the linked record is not static. The returned answer continues to reflect Traffic Manager health decisions.

Step	Portal action	What to configure
1	Browse to contoso.com	Confirm the priority 1 endpoint serves the application.
2	Stop priority 1 endpoint	Uncheck the enable endpoint config in tm-profile or stop the VM
3	Monitor Traffic Manager	Wait until the endpoint status changes to unhealthy.
4	Clear local cache	On Windows, run ipconfig /flushdns before querying again.
5	Query or browse again	Confirm the priority 2 endpoint is now selected.
6	Restore the primary endpoint	Start the endpoint and confirm it returns to online.

Why this matters for real architecture

Cleaner DNS answers

The client receives the effective endpoint result directly instead of resolving an intermediate trafficmanager.net name.

Zone-apex routing

Linked records can be used at the root domain, where DNS standards do not allow a CNAME record.

DNSSEC compatibility

Keeping resolution inside Azure DNS removes the unsigned intermediate trafficmanager.net hop from the customer response chain.

Operational type safety

Strictly typed profiles help ensure that the linked DNS record type and Traffic Manager endpoint response type remain compatible.

Preserved routing intelligence

Traffic Manager routing methods, endpoint monitoring, and automatic failover continue to determine the selected endpoint.

Preview and operational considerations

Treat the feature as preview and review the Microsoft Azure preview supplemental terms before production use.
Confirm that the DNS zone and Traffic Manager profile subscriptions have the required resource provider registration.
Plan the record type before creating the Traffic Manager profile because the strictly typed profile value cannot be changed after creation.
Remember that the record TTL is inherited from the Traffic Manager profile.
Test routing, endpoint health, DNSSEC behavior, zone-apex resolution, and rollback in a controlled environment.
Document the previous DNS record so the change can be reversed if the validation plan fails.

Conclusion

Traffic Manager linked records is a focused DNS improvement with meaningful architectural impact.

The feature keeps Azure DNS authoritative for the customer domain, preserves Traffic Manager health-aware routing, and removes the need for the client to follow an intermediate trafficmanager.net hop. That creates a cleaner DNS response path and enables scenarios that were difficult with a traditional CNAME, especially zone-apex routing and DNSSEC-protected domains.

The strongest way to evaluate the feature is through a small, observable lab: create a strictly typed profile, link an Azure DNS record, query the authoritative DNS server, and deliberately fail the primary endpoint. When the DNS response changes to the secondary healthy destination without exposing an intermediate Traffic Manager name, the value becomes immediately clear.

Final takeaway

A small change in DNS integration can simplify the client experience without giving up Traffic Manager routing intelligence, monitoring, or failover.

I hope you enjoyed it!

References

Visual note: Figures in this document were generated with Microsoft Copilot for explanatory use.

Kick Off the Fall by Building the Future: Join Microsoft's Government Agent-a-thon in Arlington, VA

Abby_Quinn — Wed, 26 Aug 2026 15:57:29 GMT

On September 29, 2026, Microsoft is launching the first event in our FY27 Frontier Innovation Series with an immersive, hands-on experience exclusively designed for Government, Defense, and Education organizations in the Washington DC region. Hosted at the Microsoft Innovation Hub in Arlington, Virginia, this collaborative workshop will help attendees gain practical skills, build working agents, and discover new ways to unlock value from Microsoft 365 Copilot.

The Next Evolution of AI is Here

Most organizations have begun exploring generative AI. Many are already seeing value with Microsoft 365 Copilot. But now the conversation is shifting toward something even more powerful: agents.

Agents can help automate recurring tasks, deliver information in context, support decision-making, and streamline everyday work. The opportunity is significant, but many organizations are still asking:

When should I use Copilot? When should I use an agent? And how do I actually build one?

This Agent-a-thon is designed to answer those questions through hands-on learning and real-world application.

Learn by Building

Rather than spending the day listening to presentations, participants will actively engage in a team-based experience focused on designing, testing, and refining agents using Microsoft 365 Copilot Agent Builder and Copilot Studio.

Working in small groups, you'll explore practical scenarios, experiment with natural language instructions, and leverage Microsoft 365 data to create solutions that support productivity and mission outcomes.

No advanced development experience is required.

Whether you're an IT leader, innovation champion, business stakeholder, or Microsoft 365 Copilot user looking to take the next step, you'll have the opportunity to learn by doing.

Real-World Outcomes, Not Just Theory

One of the reasons Agent-a-thons have become so popular is their focus on tangible outcomes.

The goal isn't simply to learn about agents.

The goal is to leave with ideas, prototypes, and repeatable patterns that can be brought back to your organization.

Throughout the day you'll discover:

How to decide when Microsoft 365 Copilot is the right solution and when agents provide additional value

How to design personal productivity agents using natural language and organizational knowledge

How to identify opportunities to streamline workflows and reduce repetitive work

How to collaborate with peers from other government organizations and learn from their approaches

How to create reusable solutions that can inspire future AI initiatives

By the end of the event, you'll understand not only what agents are, but how they can be applied to solve meaningful challenges within your own environment.

Connect with Peers Across Government

One of the most valuable aspects of this experience is the opportunity to collaborate with others who are navigating similar AI adoption journeys.

Participants will share ideas, exchange lessons learned, receive feedback from Microsoft experts, and discover new approaches that can accelerate success.

Often the most impactful innovation comes from seeing how others are solving similar problems.

Why You Should Attend

If you're:

Curious about agents and agentic AI
Looking to get more value from Microsoft 365 Copilot
Hoping to improve prompting and AI solution design skills
Exploring ways to accelerate productivity across your organization
Seeking practical, low-code and no-code approaches to innovation

Then this event is for you.

Whether you're just beginning your AI journey or looking to move from experimentation to implementation, the Agent-a-thon offers a unique opportunity to gain hands-on experience and build confidence with emerging AI capabilities.

Event Agenda

9:00 AM – 9:30 AM
Arrival & Networking

9:30 AM – 10:00 AM
Generative AI and Government Use Cases

10:00 AM – 10:45 AM
Microsoft 365 Copilot & Agents Masterclass

11:00 AM – 3:00 PM
Team-Based Labs, Building, and Hacking

3:00 PM – 4:00 PM
Team Presentations, Showcase & Awards

Be Part of the Future of Government Innovation

Organizations that succeed with AI won't simply consume technology. They'll learn how to shape it around their mission, people, and processes.

This Agent-a-thon provides a safe, collaborative, and highly interactive environment to do exactly that.

Join Microsoft and your government peers as we kick off FY27 by exploring what's possible with Microsoft 365 Copilot, Agent Builder, and Copilot Studio.

Come curious. Leave inspired. Leave with an agent.

We can't wait to see you in Arlington. 🚀

Register here.

M365 Copilot Government Roadshow is BACK! Kicking off in Atlanta with our First Ever Agent Bootcamp!

Abby_Quinn — Wed, 26 Aug 2026 15:56:45 GMT

This is exactly the kind of event government customers have been asking for: less theory, more building. If you've been hearing about AI agents and wondering how to turn the promise into practical outcomes for your agency, this is your opportunity to roll up your sleeves and create something real.

M365 Copilot Government Roadshow: Agent Bootcamp

Build Your First AI Agent in a Day

The next wave of AI innovation isn't just about asking better questions. It's about creating intelligent agents that can help automate work, streamline processes, and support mission outcomes. On September 29 in Atlanta, Microsoft is bringing together government leaders, innovators, and Microsoft 365 Copilot users for an immersive, hands-on Agent Bootcamp designed to help you go from concept to working prototype in a single day.

This isn't another presentation-heavy event where you sit through hours of slides. It's a collaborative build experience where you'll learn, design, experiment, and create alongside Microsoft experts and government peers facing many of the same challenges as your organization.

Why Agents Matter

Across government, organizations are exploring how AI can help reduce administrative burden, improve employee productivity, accelerate decision making, and enhance citizen services. Agents represent one of the most exciting opportunities to achieve these outcomes.

Think of agents as purpose-built AI assistants that can be grounded in your organization's knowledge, follow specific instructions, and help users complete real work. Whether you're looking to streamline internal processes, improve knowledge access, or support frontline teams, agents can play a critical role in your AI strategy.

The question is no longer if agents will impact government operations.

The question is: How do you get started?

That's exactly what this bootcamp is designed to answer.

What You'll Experience

Throughout the day, you'll be guided through a structured journey that combines learning with hands-on application.

You'll discover:

What agents are and how they differ from traditional AI experiences
How to identify high-value government use cases
Best practices for instructions, grounding, and knowledge sources
How to create agents using Microsoft 365 Copilot Agent Builder and Copilot Studio Lite
Approaches for testing, refining, and scaling agent solutions
Most importantly, you'll apply everything you learn immediately.

From Idea to Prototype

The heart of the event is an interactive build sprint where participants work in teams to solve real government-inspired scenarios.

You Will:

Explore practical use cases

Collaborate with peers

Design an agent solution

Build and test a working prototype

Share your work during team showcases

Learn from the creativity and innovation of other groups

By the end of the day, you'll leave with more than knowledge. You'll leave with hands-on experience, actionable frameworks, and a tangible example of what's possible when you combine government expertise with AI innovation.

No Coding Required

One of the biggest misconceptions about agents is that they require deep technical expertise.

They don't!

Whether you're a business leader, mission owner, IT professional, innovation champion, or existing Microsoft 365 Copilot user, this workshop is designed for you. The focus is on problem solving, design thinking, and practical application rather than software development.

If you can identify a challenge worth solving, you can participate in building an agent.

Who Should Attend?

This event is ideal for:

Government IT leaders

Digital transformation teams

Program and mission owners

AI champions

Business decision makers

Technical decision makers

Microsoft 365 Copilot users interested in expanding into agents

Whether your agency is just beginning its AI journey or already seeing success with Microsoft 365 Copilot, you'll gain valuable insights and practical experience that can help shape your next phase of adoption.

Event Details

September 29, 2026

9:00 AM – 4:00 PM EDT

Microsoft Atlanta South Building

200 17th Street NW

1st Floor MPR 1.1A

Atlanta, GA 30363

Agenda Highlights

Agent Foundations & Live Demonstration

Team Formation & Use Case Selection

Hands-On Agent Build Sprint

Team Showcase & Awards

Take the Next Step in Your AI Journey

The most successful organizations are moving beyond AI exploration and into AI creation. This bootcamp provides a unique opportunity to learn directly from experts, collaborate with peers, and experience firsthand how agents can support government missions.

Seats are intentionally limited to ensure an interactive, high-value experience for every attendee.

If you've been waiting for the right moment to start building with agentic AI, this is it.

Registration closes September 27, 2026.

Stop paying for idle VMs — safely: ringed start/stop waves for your Azure estate

zmustafa — Wed, 26 Aug 2026 14:49:01 GMT

Turning non-production VMs off at night is the easiest Azure saving there is. Doing it without causing an outage is the part nobody writes about. This is an open-source scheduler that takes that saving as an application, in ordered rings, with two independent safety gates between you and a real power action — and it runs entirely in your own subscription.

It is 3am. Somewhere in your subscription, a dev estate is fully powered and doing absolutely nothing. Nobody has logged in since 18:40. The bill does not care.

The fix looks trivial for about ten minutes. Write a script, stop everything at 19:00, start everything at 07:00, collect the applause. Then someone runs it against an environment where the database tier and the app tier are just two more entries in the same resource group, the app tier comes up first, spends four minutes failing its health probe, and the platform team spends the morning explaining why the cost-saving initiative caused an incident.

The saving is easy. The sequencing is what breaks.

The whole application, end to end. Everything you are about to see runs against the built-in demo estate with both safety gates off — note the mock-mode banners.

Azure VM Scheduler is an open-source, MIT-licensed scheduler that models your estate the way you actually talk about it — applications hold rings, rings hold virtual machines — and fans every scheduled occurrence out into an ordered, staggered wave of per-VM actions. Starts walk the rings forward. Stops unwind them in reverse. And until you deliberately flip two separate switches, none of it touches a real machine.

Two problems, not one

The money

Deallocating a VM stops the compute meter. That is the whole trick, and it is a good one: a machine that only needs to run 07:00–19:00 on weekdays is genuinely needed for 60 of the 168 hours in a week — about 36%. The other 64% is compute you are buying and nobody is using.

Be precise about what you keep paying for, because this is where "we'll save 64%" quietly becomes a credibility problem in your next FinOps review: deallocation stops compute charges, not storage. Managed disks keep billing whether the VM is running or not, as do reserved public IP addresses and anything else with its own meter. The saving is real and it is large, but it is a saving on the compute line, not on the invoice total.

The shape

Azure already gives you several ways to turn a machine off on a timer, and they work. The catch is that they are shaped like a machine, or like a tag, and applications are not shaped like either. An application has an order. The data tier comes up before the app tier; the front door comes up last; and on the way down it all has to happen in reverse. A canary ring exists precisely so that it is the first thing to come up and the last thing to go down.

None of that is expressible as "shut this VM down at 19:00", because the unit is wrong. What a large estate needs is a scheduler whose unit of scheduling is the application, with ordered stages inside it — and that is a modelling problem, sitting on top of primitives Azure already exposes very well.

Where this fits alongside what Azure already gives you

Azure is not short of building blocks here. Azure Resource Graph will tell you what is deployed across every subscription you can see. Azure Resource Manager exposes start, deallocate and powerOff as first-class operations with proper RBAC around them. Azure Policy, Azure Automation and the platform's own auto-shutdown all sit on that same foundation. This project is not an alternative to any of that — it is a client of it. Discovery goes through Azure Resource Graph, power actions go through ARM, and identity goes through Microsoft Entra ID.

What it adds is a shape:

Tool	Designed for	Unit of action
VM / DevTest Labs Auto-shutdown	A single VM's daily shutdown time	One VM
Azure Automation runbooks	Arbitrary automation you author and maintain	Whatever you script
Start/Stop VMs v2	Scheduled, sequenced and CPU-triggered start-stop across whole scopes	Subscription, resource group or VM list
Azure VM Scheduler	Ordered, application-aware start/stop waves with per-action safety gates	Application → ring → VM

Read that table honestly. If a resource group is a good enough boundary for your estate, use the built-in option — it is less to run, less to patch and less to explain to your auditor. Plenty of dev estates genuinely are flat, and for those this project is over-engineering.

This exists for the estates where it isn't. Where boot order matters, where a canary ring means something, where "stop everything in rg-dev" would take down the one machine that runs overnight settlement, and where the blast radius of a scheduling mistake is measured in incidents rather than in dollars.

A closer look at Start/Stop VMs v2

Of the built-in options, Start/Stop VMs v2 is the nearest neighbour, so it deserves a proper description rather than a table row. It is a Microsoft-published solution you deploy into your own subscription: an Azure Functions app holding a managed identity in Microsoft Entra ID, five Azure Logic Apps that carry the schedules and call that app with a JSON payload, Azure Storage for execution metadata and queues, and Application Insights behind a shared Azure dashboard with optional email through an action group. Each action is scoped to one or more subscriptions, resource groups, or an explicit VM list — with wildcard exclusions — and machines are ordered within a scope by tagging them sequencestart and sequencestop with values from 1 to N. It also does something this project does not: AutoStop watches CPU utilisation and stops idle machines via an alert.

One thing to know if you are choosing today: Microsoft's guidance states that no further development or enhancements are planned for Start/Stop VMs v2 beyond what is needed to keep its components on supported versions. That is ordinary platform evolution, not a warning — it still deploys, it was moved to the .NET 8 isolated worker model in 2024, and it still does what it does well.

	Start/Stop VMs v2	Azure VM Scheduler
Where the schedule lives	Azure Logic Apps you deploy and manage	The application's own database, edited in a UI or over its API
How machines are grouped	Subscriptions, resource groups or a VM list, with wildcard exclusions	Application → ring → VM, inherited down the tree
How order is expressed	A sequencestart / sequencestop tag per machine, processed in ascending order	The ring's sequence, inherited by every machine in it
Where results land	Application Insights, a shared Azure dashboard and action-group email	Per-run and per-attempt records in the app, with per-attempt retry
Where it runs	Azure	Azure, or a laptop on SQLite

Three differences follow from that, and they are differences of design centre rather than of quality.

Grouping is modelled, not enumerated. A scope in Start/Stop VMs v2 is written into the payload of the Logic App that owns it. Here the scope is a thing in the model: schedule the application and every ring and machine underneath inherits it, so adding a VM to a ring puts it in the right wave with nothing else to edit and no tag to remember.

Order and overlap are properties of that model. Sequencing tags are processed in ascending order for both directions, so a reverse-order shutdown is something you author machine by machine in the sequencestop values. Here reverse is the default for stops — the wave diagram further down this post is two rows of the same list read in opposite directions. And because schedules can be inherited, they can overlap, which is why resolution is per action and nearest-wins: a property you need only once inheritance exists.

The guards are sized to the unit. When the thing you point a schedule at is an entire application, a mistake is proportionally bigger — so the stop path gets two independent gates, never_stop that a machine inherits from any ancestor rather than an exclusion list per schedule, and an exact-count confirmation. That machinery is the price of the larger scoping unit, not a criticism of a smaller one.

So: if you want CPU-triggered auto-stop, availability in the US Government cloud, a Microsoft support path, or simply nothing extra to run beyond Azure resources themselves, Start/Stop VMs v2 is the better answer. This project is for the estates where ordering, inheritance and blast radius are the hard part.

Start/Stop VMs v2 details verified against its overview documentation, July 2026.

Model the estate the way you talk about it

The hierarchy is deliberately, aggressively shallow: an application holds rings, a ring holds virtual machines, and that is the entire tree. Exactly two levels, enforced everywhere — on create, on move, on CSV import, on settings import. There is no way to end up with a ring inside a ring inside a ring, which means there is never a debate about what a given schedule actually covers.

The built-in demo estate shows the idea in about thirty seconds. Zava Commerce has three rings — Canary (one VM), Pilot (two), Production (four). Zava Analytics has Batch and Interactive. Zava Intranet has Pilot and Production. If your estate is tiered rather than ringed, name them Data, App and Web instead; the model does not care what a stage is called, only that stages are ordered.

Attach a schedule to the application and every ring inherits it. Override one ring and only that ring changes. Override a single VM and only that machine changes. That is the whole inheritance story, and it is short on purpose.

The ring board for one application. The sequence number on each ring is the whole ordering model — there is nothing else to configure.

Waves: what actually happens at 06:30

One occurrence produces one run. One run fans out into one attempt per virtual machine, ordered by ring sequence, with a configurable stagger between machines so you never hand ARM several hundred simultaneous power requests and watch it start throttling you.

Zava Commerce's start wave fires at 06:30 with a 60-second stagger. The canary machine goes first. A minute later the two pilot machines. A minute after that the four production machines, one per minute. Seven machines, seven attempts, in a defined order, each recorded.

The stop wave at 20:00 does the same thing backwards. Stops default to reverse — the last ring in the sequence is the first one down, and the canary ring, which came up first, goes down last. That default is not a convenience; it is the single most important behaviour in the product, because reverse-order shutdown is exactly what a hand-rolled script forgets.

One application, two waves. Read the stop row right to left and you have the start row — that is the point. If you only take one picture from this post, take this one.

Nearest schedule wins — per action

Here is the guarantee that makes overlapping schedules safe to live with. For each action independently, a VM is acted on by the nearest schedule that targets it: a schedule on the VM beats one on its ring, which beats one on its application. Deeper shadows shallower.

Because start and stop are resolved separately, a machine ends up with at most one effective start and at most one effective stop. Stack an application-wide 06:30 start, a ring-level 07:15 start and a per-VM 05:00 start on the same machine and it still starts exactly once, at 05:00. It cannot be started twice. It cannot be stopped twice. That property holds no matter how many schedules sit above it, which is what lets you layer overrides without keeping a mental model of the whole tree.

A wave running. Each row appears as its machine's turn comes up, in ring order, one stagger interval apart.

Every wave leaves a paper trail

Every wave is fully reconstructable afterwards. A run carries a rolled-up status — succeeded, partially_failed, failed, timed_out or cancelled — and each attempt records its own status, whether it ran in real or mock mode, a message, its attempt number, its sequence position and a correlation id. When someone asks "what happened at 06:30 on Tuesday", the answer is a page, not an investigation. Failed attempts can be retried individually, or the whole run can be retried and it will pick up only the ones that failed.

One wave, after the fact: the mode it ran in, the correlation id, the roll-up, and every machine it touched. This page is the answer to "what happened at 06:30 on Tuesday".

Recurrence you can check before you trust it

Schedules come in four flavours: one-time, daily, weekly and full five-field cron — lists, ranges, steps, month and day names, Sunday as 0. Daily and weekly are stored as friendly fields and translated to cron behind the scenes, so there is exactly one occurrence engine in the codebase rather than one engine and three special cases that disagree at the edges.

Everything is timezone-aware and stored against an IANA zone (the demo estate runs on America/New_York). Times are wall-clock: 08:00 stays 08:00 across a daylight-saving change, which is what an operations team means when they say "eight in the morning". And an occurrence that lands in a spring-forward gap — an 02:30 job on the morning the clocks jump from 02:00 to 03:00 — is skipped, not silently shifted to 03:30. That is a decision, made once, applied consistently, and covered by tests.

The preview is the point

Cron is where scheduling bugs live. Everybody has confidently written 0 0 * * 0 and then argued about which day that is.

So the editor does not ask you to trust it. On every keystroke it asks the server to describe the recurrence in plain English and return its next five occurrences — and the server answers using the same engine that will actually fire the schedule. Not a browser re-implementation of cron that agrees with the backend most of the time. The same code path. What the editor shows you is, by construction, what the scheduler will do.

Watch the right-hand panel as the cron expression is typed. That description and those five dates come back from the server on every keystroke — from the same engine that fires the schedule.

Schedules that know when to stop

A schedule can carry start_date and end_date as local calendar bounds, and a run_limit budget. When the budget is spent or the end date passes, the schedule flips to status completed and stops producing a next run — rather than sitting there looking enabled while silently never firing again, which is the failure mode that has you debugging a scheduler that is working perfectly. Manual runs do not consume the budget; only scheduler-triggered ones do.

Why you can point this at production without flinching

This is the section the rest of the post exists for. Seven guarantees, each one a mechanism rather than a promise.

The gates

Two independent gates.A real Azure start requires the globalENABLE_REAL_AZURE_STARTS setting and the target tenant's allow_vm_start permission. A real stop requires ENABLE_REAL_AZURE_STOPS and allow_vm_stop. They are entirely separate. Turning on starts does not, and cannot, enable stops.
Re-evaluated on every single attempt.The permission decision is deliberately kept apart from the expensive work of building an ARM credential. Adapters get cached because acquiring a token is costly; thedecision is remade for every attempt. Revoke a tenant's stop permission while a stop wave is halfway through and the remaining attempts stop being real — immediately, not whenever a cached adapter happens to expire.
Mock-first.Until both switches are on, every wave runs against a deterministic mock adapter that records a simulated result. You can build an entire estate, wire up every schedule, preview the next month of occurrences, run the waves, inspect the attempts and show the whole thing to your change board without touching a single machine. A fresh deploymentarrives inert — both gates ship false.

The gates are a visible posture, not a buried environment variable. This is what a fresh deployment looks like, and it is what it keeps looking like until somebody changes it.

The guards

never_stop.Set it on a VM, or on any ancestor, and that machine is removed from every stop wave and every on-demand stop. Zava Payments demonstrates it: its Production ring runs overnight settlement, so it is marked never_stop and the application has a start wave but no stop wave at all. Nothing you do to the parent schedule can pull those machines into a shutdown.
Exact-count confirmation.Selecting machines and pressingStop now stops nothing. The dialog names the tenant, states whether it will deallocate or power off, and refuses to arm until you type the machine count back. It is unglamorous and it has saved somebody's afternoon.

Eighteen machines selected, but the dialog offers to stop sixteen — two are never_stop and it says so. The confirm button stays dead until 16 is typed. Three of the seven guarantees, in one dialog.

Conflict guard.The scheduler skips an attempt whose VM already has an opposite-action attempt in flight. If a start wave is still working on a machine, a stop wave will not race it.
Read-only tenants can never power anything.A connection marked read-only refuses start and stop outright, gate or no gate, and a disabled connection refuses everything.stop_mode is deallocate by default — the one that actually stops the compute meter — with power_off available per schedule when you need the machine halted but not released.

Coverage gaps: the report you didn't know you needed

Any scheduling system's second-order failure is partial coverage. Not "the scheduler broke" — the scheduler is fine — but "eleven machines quietly fell outside it". So the overview goes looking for exactly that, in both directions:

Starts but never stops — machines with an effective start wave and no effective stop wave. These come up every morning and never go down. They are the ones burning money while everyone congratulates themselves on the savings.
Stops but never starts — the inverse, and the more expensive mistake. These go down tonight and stay down, and you find out at 09:00 tomorrow.
Stop protected — machines excluded by never_stop, listed explicitly so the exclusion is a visible decision rather than something you rediscover during an incident.

It also flags start and stop waves that overlap in time, including the stagger tail, so you find out at design time that your 20:00 stop wave for a forty-machine application is still running when the 20:30 start wave for something else begins.

Gaps in both directions, named machine by machine. The middle one — three machines that stop tonight and are never started again — is the one that ruins a Monday.

The same page carries the pre-flight readiness checks (both global gates, tenants that are read-only or missing the permission a live schedule needs, credentials approaching expiry), windowed KPIs with a previous-period delta, a fourteen-bucket trend, the next-24-hours strip, the rollout plan, an application health matrix and reliability statistics.

Getting a real estate in without a spreadsheet

Nobody types four hundred resource IDs. Three routes in.

Paste bare VM names

Give it vm-commerce-prod-01 and it resolves the subscription and resource group for you through Azure Resource Graph. If a name is ambiguous across subscriptions, it surfaces the candidates and asks you to pick rather than guessing. Duplicates are blocked. You can also browse a subscription and select machines directly, or paste full resource IDs if you already have them.

Four names in. Three are already filed and it tells you exactly which ring holds each; the fourth is unknown and gets offered a tenant to resolve against.

Import VMs

Point it at a UTF-8 inventory CSV and it runs a validating preview that reports exactly what would be created, updated and skipped before anything is written. The preview is bound to an encrypted, expiring token, and the commit is atomic — a stale or tampered preview is rejected rather than half-applied. The simplest valid file is a single column of VM names.

Nothing is written during a preview. The commit is all-or-nothing, and a preview that has gone stale is refused rather than partly applied.

Who gets to press the button

Access control is capability-based, and it is enforced on the API rather than hidden in the UI.

Users hold roles directly or through access groups; their effective permissions are the union of both, resolved once per request. Five roles ship built in — admin, operator, auditor, viewer, noaccess — and they are re-seeded on every start, so a capability added in a new version is never left unusable on an existing deployment. Custom roles are free-form on top.

Sign-in is local, or SSO, or both. SSO is multi-provider: any number of OIDC issuers configured from their discovery documents (Microsoft Entra ID is just the case where the issuer is derived from a directory id), and SAML 2.0 verified with signxml — where the code reads only the signed subtree the library returns, never the raw document, which is how a whole family of SAML signature-wrapping attacks stops being interesting.

Three walls are enforced for every request: a noaccess allowlist, a forced-password-change allowlist, and a database-backed per-IP brute-force throttle that catches one attacker spraying many usernames rather than only many attempts at one account. Two lock-out guards return 409 rather than letting you strand yourself: you cannot remove the last enabled account that can manage users, and you cannot disable local sign-in when no SSO provider is enabled.

Every action lands in an audit log.

Users, roles, access groups, live sessions, sign-in policy and the SSO providers — one page, one permission to reach it.

Where it runs

One container image. The built single-page app is served by FastAPI at the same origin, so there is no CORS story, no second container and no reverse-proxy configuration to get wrong. The static mount is registered last, so it can never shadow the API.

The Deploy to Azure button provisions, in your subscription: a Container App running the public image, Azure Database for PostgreSQL flexible server (Burstable B1ms), an Azure Files share mounted at /app/.data for the encryption key and connection registry, a Container Apps environment and a Log Analytics workspace. Replicas are pinned to one, because the scheduler is in-process by design. Setting privateNetworking at create time injects the environment into a VNet and puts both the database and the storage account behind Private Endpoints with public access disabled.

Everything one deployment creates. The private-networking switch is create-time only — an existing public deployment has to be redeployed, not flipped.

Connecting Azure without storing a secret

The Container App gets a system-assigned managed identity. Grant it Reader on the scope you want to manage plus start / deallocate / powerOff on the target VMs, add a tenant in the app using the default_chain auth method, and no credential is stored anywhere at all. Where you do store client secrets, they are Fernet-encrypted at rest and never leave the host.

Two tenants, one writable and one read-only, both authenticating as the host identity. The per-tenant permissions are the second half of every gate.

Or run the whole thing on a laptop against SQLite. DATABASE_URL picks the engine; nothing else about the application changes.

How it's judged

The claims above are mechanisms, and each one is testable:

A VM is never started twice nor stopped twice, whatever the schedule overlap — resolution is per action, and the nearest schedule wins.
The recurrence preview is produced by the same engine that fires the schedule, so what the editor shows is what the scheduler does.
An occurrence in a DST spring-forward gap is skipped, never silently shifted.
A real Azure power action requires two independent switches, re-checked on every attempt — so revoking a permission halts a wave already in progress.
never_stop is honoured by scheduled stops and on-demand stops alike.
Every wave is reconstructable after the fact: run status, per-VM attempt, mode, message, attempt number, sequence position and correlation id.
A fresh deployment cannot touch Azure until a human deliberately arms it.

What it deliberately isn't

Worth knowing before you invest an afternoon:

Single-replica by design. The scheduler runs in-process and the Bicep pins replicas to 1/1. That is a conscious trade — no leader election, no distributed lock, no split-brain — but it does mean this is not a highly-available control plane.
Virtual machines only. Not scale sets, not AKS node pools, not App Service plans, not SQL.
Not a cost tool. It will not price your savings, forecast them or chargeback them. It turns machines off; your existing cost tooling reports on the result.
Two levels, permanently. If you need a five-deep hierarchy, this is the wrong shape and no amount of configuration will change that.

Try it

The fastest honest test is to run it against nothing. Deploy it with the button below — it arrives with both action gates off, so it physically cannot touch a machine — then open Settings → Demo data and load the sample Zava estate: four applications, nine rings, eighteen virtual machines and seven start/stop waves, complete with a never_stop ring and an application that starts but never stops, so the coverage-gap detection has something to find. It loads in one click and removes exactly what it created.

Build your real rollout on top of that, rehearse every wave against the mock adapter, and only then decide whether to arm anything.

Let GitHub Copilot do the setup

Open an empty folder, put Copilot Chat in agent mode, and hand it the repository:

Clone https://github.com/zmustafa/AzureVMScheduler into this folder, set it up for local development, then start the backend and the frontend.

That is enough. The repository ships a README with the setup steps and VS Code tasks for both servers, so the agent has something to follow rather than a command line to invent. The API comes up on 127.0.0.1:8000 and Vite on 127.0.0.1:5173 proxying /api to it; with no DATABASE_URL set, the whole thing runs on a SQLite file under .data/ with nothing else to install. Sign in as admin, change the password when it makes you, and load Settings → Demo data.

Repository: https://github.com/zmustafa/AzureVMScheduler
Container image: https://hub.docker.com/r/zmustafa/azure-vm-scheduler
Licence: MIT — issues and pull requests welcome

If you try it, I would genuinely like to hear which part of the safety model you found excessive and which part you found insufficient. That is the argument worth having.

Original post.

This is a community open-source project and is not affiliated with or endorsed by Microsoft.

Navigating the Azure Databricks CSP Mandate

anishekkamal — Wed, 26 Aug 2026 14:47:42 GMT

Mandatory Deadline: The Compliance Security Profile (CSP) becomes mandatory for processing HIPAA, HITRUST, and IRAP regulated data on Azure Databricks by September 1, 2026.
Permanent Change: Enabling CSP is an irreversible, workspace-level configuration. Reversion requires deleting and recreating the workspace, emphasizing the need for meticulous pre-enablement testing.
Architectural Impact: CSP introduces significant changes, notably requiring VNet encryption which can disrupt existing on-premises connections, and disabling partner-powered AI features by default.

The evolving regulatory landscape demands rigorous security and compliance measures for data processing in the cloud. For organizations leveraging Azure Databricks, the introduction and upcoming mandatory enforcement of the Compliance Security Profile (CSP) represent a pivotal shift. This profile is not merely a feature toggle; it's a foundational architectural commitment designed to harden your Databricks environment against stringent compliance standards. Understanding its implications and preparing proactively is crucial to ensure uninterrupted operations and continued regulatory adherence.

Why CSP Matters

The Compliance Security Profile (CSP) is a robust, platform-hardening configuration applied at the workspace level within Azure Databricks. Its primary objective is to facilitate adherence to a broad spectrum of compliance standards by enforcing additional monitoring, utilizing hardened compute images, ensuring inter-node encryption with specific instance types, and implementing stricter controls across Databricks workspaces. This provides a secure baseline for the data plane, incorporating enhanced security monitoring capabilities.

For organizations handling sensitive data, such as Protected Health Information (PHI) under HIPAA, or data subject to HITRUST and IRAP regulations, CSP is not just beneficial—it is becoming mandatory. Starting September 1, 2026, all Azure Databricks workspaces processing data under HIPAA, HITRUST, or IRAP will be required to have CSP enabled. Failure to comply will impede the ability to process such regulated workloads.

Conceptual diagram illustrating data protection and complianceonDatabricks.

Key Compliance Standards Supported by CSP

The compliance security profile supports or is required for processing data under a wide array of global and industry-specific compliance standards. This comprehensive coverage underscores its importance for organizations operating in highly regulated sectors:

C5
CCCS Medium (Protected B)
FedRAMP
HIPAA (Health Insurance Portability and Accountability Act)
HITRUST (Health Information Trust Alliance)
IRAP (Infosec Registered Assessors Program)
ISMAP
Korean Financial Security Institute (K-FSI)
PCI-DSS (Payment Card Industry Data Security Standard)
TISAX
UK Cyber Essentials Plus

Understanding CSP's Architectural Implications

One of the most critical aspects of CSP to grasp is its permanence. Once enabled on a workspace, or once any regulated data has been processed within a CSP-enabled workspace, the profile cannot be simply toggled off. To revert to a non-CSP state, you would need to delete the existing workspace and provision a new one without the CSP activated. This irreversible nature necessitates a thorough planning and validation phase before implementing CSP in production environments.

Cost Considerations: The Enhanced Security & Compliance Add-on

Enabling CSP automatically activates the "Enhanced Security and Compliance add-on" billing, which will incur additional costs. Organizations must factor these increased expenditures into their budgeting and financial planning. Reviewing the Azure Databricks pricing page is essential to understand the full financial impact.

Networking Challenges: VNet Encryption and On-Premises Connectivity

A significant "gotcha" highlighted in various sources is the impact of CSP on network architecture. CSP enforces stricter network security policies, often requiring VNet encryption. If your Azure Databricks workspaces utilize VNet Injection and connect to on-premises systems via ExpressRoute or VPN, this VNet encryption can potentially disrupt existing connections. This necessitates close collaboration with networking teams to verify compatibility, test connectivity in pre-production environments, and potentially reconfigure network components such as routing tables, security groups, or firewall rules.

Feature Impact: AI Assist and Public Preview Limitations

To maintain stringent compliance boundaries, CSP workspaces alter default behaviors and restrict certain features:

Partner-powered AI Features: By default, "Partner-powered AI features," which include some AI-powered coding assistants like Genie Code, are disabled in CSP workspaces. While Databricks-hosted models may power Genie Code in CSP workspaces for specific compliance standards (e.g., HIPAA, FedRAMP Moderate, PCI-DSS, HITRUST) in certain regions, organizations relying on these features should assess alternatives or plan for explicit enablement, understanding the potential constraints.
Preview Features: Only a limited set of Public Preview features are supported in CSP workspaces. Workloads dependent on unsupported preview features may face limitations or require adjustments.

Preparing for the CSP Mandate

To ensure a smooth transition to CSP, a detailed and phased approach is recommended. This involves thorough assessment, strategic planning, rigorous testing, and clear communication across various teams.

The radar chart above visualizes the current readiness level (Pre-CSP Readiness) versus the desired target state (Post-CSP Target State) across critical preparation areas for the Compliance Security Profile. It underscores the significant effort required in areas like network reconfiguration and thorough testing to achieve full compliance and operational stability.

Inventory and Assessment
- Identify Regulated Workspaces: Catalogue all existing Azure Databricks workspaces. Determine which ones currently process, or are planned to process, data subject to HIPAA, HITRUST, or IRAP.
- Review Data Pipelines: Map out all data ingress and egress points for these identified workspaces, including connections to on-premises data sources, other cloud services, and external APIs. This helps identify potential network impacts.

Network Architecture Review

- Evaluate VNet Encryption: If your workspace uses VNet Injection and has private connectivity to an on-premises network (e.g., via ExpressRoute or VPN), verify that these connections will remain functional under CSP's new encryption requirements.
- Plan for Reconfiguration: Allocate resources for potential network re-architecting, including updating routing tables, security groups, or firewall rules to accommodate encrypted traffic flows.

Feature Impact Analysis

- AI & Assist Features: Assess the reliance of your development teams on "Partner-powered AI features" like Genie Code. Determine if Databricks-hosted alternatives meet their needs or if explicit enablement (and its associated compliance considerations) is required.
- Preview Features: Consult official documentation to check for limitations on Public Preview features in CSP workspaces and plan accordingly if your workloads depend on any unsupported features.

Data Handling Responsibilities

- Sensitive Information Fields: Implement policies to prevent sensitive data from being entered into customer-defined input fields (e.g., workspace names, compute resource names, tags, job names, URLs), as these might be stored or processed outside the compliance boundary.
- Encryption at Rest: Verify that all data containing PHI or other regulated information is encrypted at rest in any storage location interacting with Databricks, including workspace storage accounts.

Execution and Validation Plan

- Staging Environment First: Always enable CSP on a clone of your production environment in a staging subscription before rolling out to production.
- Thorough Testing: Conduct comprehensive end-to-end tests on the staging workspace to validate data pipelines, network connectivity, user workflows, and performance.
- Enablement Method: Choose the appropriate tooling for enablement—Azure Portal, Azure CLI, PowerShell, ARM templates, or Terraform—to ensure consistency and automation.

Detailed Enablement Methods

Enabling CSP can be done for both new and existing workspaces using various Azure management tools. Regardless of the method chosen, it's crucial to select the applicable compliance standards during enablement.

Enabling CSP on New Workspaces

For new workspaces, the compliance security profile settings at the account level can control whether it's enabled by default. Account administrators can enable CSP individually during deployment using:

Azure Portal
Azure CLI (e.g., az databricks workspace create --enable-compliance-security-profile --compliance-standards '["HIPAA"]')
PowerShell
ARM templates
Terraform (using the databricks_compliance_security_profile_workspace_setting resource)

Enabling CSP on Existing Workspaces

Existing workspaces can also have CSP enabled through similar methods:

Azure Portal: Navigate to Workspace > Settings > Security & compliance > Enable compliance security profile; select standards and save.
Azure CLI / PowerShell / ARM templates / Terraform: Utilize these tools to modify the workspace settings and enable CSP, specifying the relevant compliance standards.

Understanding Key Technical Considerations

Beyond the general planning, specific technical details must be understood and addressed to ensure a successful CSP implementation.

Account-Level Requirements

Before enabling CSP, ensure your Databricks account meets these prerequisites:

Your Databricks account must include the Enhanced Security and Compliance add-on.
Your Databricks workspace needs to be on the Premium pricing tier.

Security Configuration Table

The table below summarizes critical security configuration aspects influenced by CSP:

Security Aspect	Pre-CSP State	Post-CSP State	Preparation Action
Compute Images	Standard images	Hardened compute images	No direct action required, platform handles.
Inter-node Encryption	Configurable/Optional	Enforced (for specific instance types)	Ensure use of supported instance types.
Network Encryption (VNet)	Standard VNet configuration	VNet encryption required	Test on-premises connectivity, reconfigure network routes/firewalls if needed.
AI Assist Features	"Partner-powered AI features" typically enabled	"Partner-powered AI features" disabled by default	Assess reliance, plan explicit enablement or alternative Databricks-hosted models.
Monitoring	Standard monitoring	Enhanced security monitoring, additional agents	Familiarize with new monitoring capabilities and data.
Compliance Scope	Self-managed controls	Platform-enforced controls for selected standards	Select appropriate compliance standards during CSP enablement.

To further understand the broader context of security and compliance on Databricks, including how CSP fits into the larger ecosystem, watch this video from Databricks: Databricks Security & Compliance

References

Compliance security profile: https://learn.microsoft.com/en-us/azure/databricks/security/privacy/security-profile
Configure enhanced security and compliance settings: https://learn.microsoft.com/en-us/azure/databricks/security/privacy/enhanced-security-compliance
HIPAA, Azure Databricks, Microsoft Learn: https://learn.microsoft.com/en-us/azure/databricks/security/privacy/hipaa
What is Azure Virtual Network encryption: https://learn.microsoft.com/en-us/azure/virtual-network/virtual-network-encryption-overview

Connectivity Errors caused by slow logins in a Microsoft Fabric SQL Database

Tancy — Wed, 26 Aug 2026 14:04:18 GMT

By Tanayankar Chakraborty & Sukhwant Kaur

Issue

A customer recently reported that their Azure App Service intermittently cannot establish a TDS connection to its Microsoft Fabric SQL database within 15 seconds. Also, Review Queue and Worklists returned HTTP 500 after approximately 15 seconds.

This occurred before any SQL statement executed and was not an Entra login/permission error.

Error

Here is the error generated in the App service logs:

At 2026-07-29 22:42:15 UTC, App Insights recorded:

ConnectionError / ETIMEOUT: Failed to connect to the Fabric SQL endpoint in 15000ms.

GET /api/review/queue returned HTTP 500 in 15,100ms.

Troubleshooting

As stated in our public documentation, throttling by Fabric platform is expected behavior (requests getting delayed by 20 seconds for initial throttling, and requests getting rejected outright if it gets worse) if usage is heavy, especially on a heavy SQL workload on workspace/capacity shared with other artifacts, with a small capacity size (F4, etc).

Assuming that's what the issue is, general guidance is to:

Decrease the max vCore limit on the affected SQL DB to limit the amount of CUs it can consume , but obviously there would be a performance tradeoff here though, and throttling can still be caused by other items in the workspace/capacity

Increase the capacity (F16+, etc)

Fabric allows operations to temporarily use more compute than the provisioned SKU through bursting and distributes that usage over future capacity through smoothing. When smoothed usage exceeds the capacity's available future compute, the excess becomes carry-forward usage. Fabric then progressively protects the capacity by delaying or rejecting new operations.

Workaround

### Mitigate Active Throttling

Microsoft's documented mitigation options include:

- Wait for the capacity to self-heal while unused capacity burns down carry-forward usage.

- Temporarily increase the F SKU size. The larger SKU provides more idle capacity and accelerates burndown.

- Pause and resume the capacity. This creates a billing event for accumulated future usage, and the resumed capacity starts with zero future usage.

- Consider capacity overage billing where appropriate. Microsoft documents this option as being billed at three times the normal capacity rate.

Any pause/resume or billing change should be reviewed by the capacity administrator for operational and cost impact.

### Improve Application Resilience

- Use bounded retry with jitter for transient connection failures where the operation is safe to retry.

- Align SQL connection timeout and retry settings with the application's end-to-end request budget.

- Capture UTC timestamps, connection/request IDs, operation names, and SQL error numbers for future incidents.

Monitoring

It is recommended to monitor and right size the capacity:

Use the Microsoft Fabric Capacity Metrics app to monitor:

- Utilization by workload and item

- Interactive and background CU consumption

- Carry-forward and burndown

- Throttling events and rejected operations

- Minutes to burndown

A capacity that is throttled might appear in the Fabric Capacity Metrics app as shown below:

In the Capacity Metrics app:

1. Use the **Utilization** chart and drill through to individual timepoints to identify operations contributing to overages.

2. Use the **Throttling** chart to determine whether an overage has crossed a throttling threshold.

3. Use the **Overages** tab to review carry-forward, additions, burndown, and cumulative utilization.

4. Configure capacity alerts so administrators are notified when provisioned CU usage reaches 100%.

### Reduce Peak Concurrent Usage

- Stagger high-CU SQL and Data Integration operations where possible.

- Review expensive or highly concurrent SQL database operations.

- Distribute workloads across capacities if sustained demand requires it.

- Use historical Capacity Metrics data to select a SKU with sufficient peak headroom.

References

Understand capacity throttling and smoothing - Microsoft Fabric | Microsoft Learn

Surge protection - Microsoft Fabric | Microsoft Learn

Billing and Utilization Reporting - Microsoft Fabric | Microsoft Learn

Control Compute Usage - Microsoft Fabric | Microsoft Learn

What is the Microsoft Fabric Capacity Metrics app? - Microsoft Fabric | Microsoft Learn

Set alerts on Fabric capacity overview events in Real-Time hub - Microsoft Fabric | Microsoft Learn

Customer review: Earth Knowledge enables compliance and prioritized action for a changing climate.

Nikhil_Viswanathan — Wed, 26 Aug 2026 13:00:00 GMT

Earth Knowledge's Integrated Planetary Intelligence, a cloud-native platform published to Microsoft Marketplace, models climate and natural impacts for government and private sector organizations. By analyzing cascading and interrelated impacts globally, Integrated Planetary Intelligence transforms complex, multi-dimensional data about the planet and the environment into actionable data enabling decision-makers to assess current and future risks and make choices that are better for their organizations and the planet. Staffed by world-class scientists, engineers, and analysts, Earth Knowledge guides organizations toward reduced risk and optimized operations.

Microsoft Marketplace interviewed Karen Lynn, VP Sustainability & EHS Management Systems at Eaton, a global power management company headquartered in the United States, to learn what she had to say about the platform.

What do you like best about Earth Knowledge Integrated Planetary Intelligence?

The solution is technically rigorous and is supported with knowledgeable technical and scientific personnel. The Earth Knowledge (EK) team is very willing to answer questions and explain the results.

How has the product helped your organization?

The solution has enabled Eaton to meet our voluntary and regulatory requirements related to climate modeling and physical risk to our company locations. The data also then helps us prioritize actions needed to prepare for a changing climate.

How are customer service and support?

Customer service and support are excellent. The EK team is always responsive. Assistance is also thoughtful and technically supported.

Any recommendations to other users considering Earth Knowledge Integrated Planetary Intelligence?

The model is technically rigorous and can be used at various levels of detail based on subscription level.

What is your overall rating for this product?

5 out of 5 stars.

Cloud marketplaces are transforming the way businesses find, try, and deploy applications to help their digital transformation. Learn more about Microsoft Marketplace and find ways to discover the right application for your business needs.

ICYMI: New Azure SQL Foundations video series with GitHub samples

Anna Hoffman — Wed, 26 Aug 2026 12:41:16 GMT

Bob Ward and I recently released a series of videos, read more in the original blog post or go directly to the series on YouTube.

The Azure SQL Database Foundations series are four videos that take you from your first Hyperscale database to AI features running against your own operational data. We also included how to assess and migrate (with AI and skills!) to Hyperscale in the first place, and the common optimizations you should consider. Every episode ships with a repo, so you can follow along in your own environment instead of watching someone else’s terminal.

Streamlining Enterprise Collaboration - ACP - SharePoint Partner Spotlight

VesaJuvonen — Wed, 26 Aug 2026 09:53:02 GMT

Organizations continue to look for faster ways to modernize intranets, improve internal communications, and increase employee engagement. Partners in the Microsoft ecosystem are helping customers accelerate that journey by extending SharePoint and Microsoft 365 with ready-to-deploy solutions that deliver immediate business value.

As organizations increasingly adopt Microsoft 365 Copilot, a well-structured and engaging intranet becomes even more important. High-quality content, clear information architecture, and effective content management help employees get greater value from AI-powered experiences.

This partner showcase highlights ACP, one of the leading IT service providers in Austria, Germany, and Switzerland.

About ACP

ACP is one of the leading IT service providers in Austria, Germany, and Switzerland. As a leading technology and digital transformation partner, ACP delivers modern IT solutions, cloud services, digital workplaces, cybersecurity, and managed services to businesses and public sector organizations. With more than 2,600 specialists and a strong local presence, ACP helps customers unlock innovation and turn technology into measurable business value.

Within ACP, the SCOPE team (SharePoint & Copilot Experts) serves as the AI-first center of excellence for Microsoft 365, SharePoint Online and Copilot. Through consulting, implementation, training, adoption programs, and product innovations, the team enables organizations to become future-ready and achieve sustainable long-term success. With more than 15 years of experience helping organizations build digital workplaces with SharePoint, the SCOPE team has helped organizations build intelligent digital workspaces, intranets, and business solutions that deliver real value - turning customer-specific use cases into practical and impactful solutions.

The ACP Intranet Accelerator Kit

To address recurring needs and challenges, ACP transformed years of project experience into a scalable product. In 2024, the company launched the ACP Intranet Accelerator Kit, a subscription-based add-on for SharePoint Online that combines the most requested intranet capabilities into a ready-to-use solution.

The ACP Intranet Accelerator Kit extends SharePoint Online with a comprehensive set of configurable capabilities designed to enhance the employee experience while reducing implementation effort and cost. Instead of building custom solutions from scratch, organizations can quickly deploy proven functionality that addresses common intranet requirements while staying aligned with Microsoft 365 governance and standards.

The solution improves several key areas of the intranet experience, including:

Personalized information delivery through synchronized user profile data
Enhanced navigation and content discoverability
Extended search and filtering capabilities
Flexible design and content presentation options
Ready-to-use intranet components for common employee scenarios
Improved content management experience for editors and site owners

These capabilities help organizations accelerate intranet modernization while reducing implementation complexity and cost. Organizations can rapidly introduce capabilities such as people directories, event hubs, FAQs, wikis, targeted news experiences, quick links, polls, and employee profile enhancements — without extensive custom development.

For employees, this means faster access to relevant information, better discoverability, and a more engaging digital content experience. For content owners and communication teams, it means greater flexibility, lower maintenance effort, and the ability to launch valuable intranet experiences significantly faster.

The ACP Intranet Accelerator Kit reflects ACP's broader innovation approach: transforming real customer challenges into reusable solutions that scale across organizations. By combining Microsoft SharePoint with practical product innovation, ACP helps customers accelerate their digital workplace journey while continuously benefiting from ongoing enhancements, support, and product evolution.

Watch the showcase

In this SharePoint Partner Showcase, ACP demonstrates how organizations can rapidly enhance SharePoint Online with proven intranet capabilities that improve communications, employee engagement, and knowledge sharing while reducing implementation effort. By optimizing corporate communications, streamlining document workflows, and boosting employee engagement, ACP helps enterprises maximize their Microsoft 365 investment and drive long-term business value.

Key highlights covered in this video:

Modern Corporate Communications – How to centralize news, announcements, and leadership messaging across distributed and hybrid teams.
Intranet Optimization – Building engaging, targeted communication sites and hub portals that eliminate information silos.
Enhanced Productivity & Collaboration – Streamlining daily workflows, content governance, and knowledge management within Microsoft 365.
Business ROI & Scalability – Accelerating deployment with proven intranet capabilities while maintaining flexibility for future growth.

Whether you're looking to upgrade your legacy intranet, improve internal comms engagement, or build a scalable employee portal, see how ACP delivers tailored solutions aligned with your core business objectives.

Agenda:

00:00 – Introduction
03:18 – Live demo
12:47 – Discussion

When asked where Microsoft could further invest for partners and customers, ACP highlighted two key areas:

Improved predictability and governance controls for new feature rollouts to support organizational change management.
Greater opportunities to integrate Copilot capabilities directly into custom SharePoint solutions and experiences.

Presenters

In this SharePoint Partner Showcase, Vesa Juvonen (Microsoft) sat down with Julia Kaufmann-Hailu (ACP) and Martin Ettl (ACP) to discuss ACP's SCOPE offerings and how the ACP Intranet Accelerator Kit helps organizations turn SharePoint Online into a modern, engaging digital workplace.

Julia Kaufmann-Hailu (ACP) – LinkedIn
Martin Ettl (ACP) – LinkedIn
Host: Vesa Juvonen (Microsoft) – LinkedIn

Learn more about ACP

Microsoft resources

Microsoft 365 and SharePoint provide powerful out-of-the-box capabilities that can be extended and tailored to your user experience goals using no-code, low-code, or pro-code approaches. This flexibility enables customers to configure and build unique and powerful experiences at Microsoft 365 for corporate communications, employee experiences, and business processes.

Microsoft SharePoint
SharePoint 25th anniversary announcements - new AI features and more
SharePoint help & learning
Intelligent intranet overview
SharePoint Framework provides a secure, modern way to build custom and AI-enhanced experiences in SharePoint and Microsoft 365.
SharePoint Embedded enables building solutions outside of Microsoft 365 with a custom user interface, but continues storing files in SharePoint.
Microsoft Community Learning YouTube channel for videos and guidance

Showcase your solution

Do you have offerings or products built with SharePoint and would be interested in creating a similar showcase video and blog post with Microsoft? Please fill in the following form to get connected with us: https://aka.ms/sharepoint/partner/showcase.

The Microsoft 365 ecosystem continues to thrive through partners who innovate, build solutions that solve real business challenges, and share their expertise with customers around the world.

Windows Update Failures in Japanese-Localized Azure VM SQL2022-WS2022 Images: Cause and Workaround

Yuki_Kitazawa — Wed, 26 Aug 2026 07:45:58 GMT

This blog post is primarily targeted at customers in the Japan region and is therefore written in Japanese as well. If you would like to read it in English, please use the link Jump to English below to navigate directly to the English section.

こんにちは、 SQL Server サポートチームです。

Azure 上で SQL Server を含むマーケットプレイスイメージ（SQL2022-WS2022）からデプロイした OS を日本語化した場合、2026 年 5 月以降の Windows Update などによる OS更新プログラムの適用が 0x800f081f や 0x800f0831 で失敗する事例について解説します。

■事象

OS更新プログラムの適用が以下のようなメッセージにより失敗します。

エラー："更新プログラムのインストール中に問題が発生しましたが、後で自動的に再試行されます。この問題が引き続き発生し、 Web 検索やサポートへの問い合わせを通じて情報を集める必要がある場合は、次のエラーコードが役立つ可能性があります。: (0x800f081f) "

※ 表示されるエラーコードは、0x800f0831 の場合もあります。

■原因

本問題は、マーケットプレイスイメージの SQL Server を含むイメージの問題により発生する場合があります。

最新のイメージでは問題を修正済みのため、現時点で新規にデプロイしたサーバーに対しては関連しません。

■確認方法

C:\Windows\Logs\CBS\cbs.log に以下のような amd64_microsoft-hyper-v-ram-parser.resources_31bf3856ad364e35_10.0.20348.1_ja-jp_531076835e0e2f98.manifest が存在しないという警告が出力されている場合、本事象に合致すると判断できます。

//参考

C:\Windows\Logs\CBS\cbs.log

2026-05-19 12:11:57, Info CBS Exec: Warning - Manifest doesn't exist for: \\?\C:\Windows\CbsTemp\31254331_3251161130\Windows10.0-KB5087545-x64.cab\amd64_microsoft-hyper-v-ram-parser.resources_31bf3856ad364e35_10.0.20348.1_ja-jp_531076835e0e2f98.manifest

■対応手順

＃＃注意事項＃＃

・事前に Azure Backup で Azure VM のバックアップを取得し、作業前の状態にリストアできる状態にします。

・当該作業時、 SQL Server サービスは事前に停止状態（スタートアップ種類を無効）にします。作業が完了した後で、起動状態（スタートアップ種類を自動や自動（遅延開始）など元の設定）にします。

・日本語の言語パッケージの再追加後は更新プログラムの再適用を行うまでは決して OS を再起動しないようにします。

　言語リソースの追加直後は OS のバイナリと追加された言語リソースとでバージョンが不一致のため、そのまま再起動を行った場合、リモートデスクトップ接続が行えなくなるなどの問題が発生する可能性があります。

＃＃＃＃＃＃＃＃

1.日本語の言語パッケージの削除

-----------------------------------

1-1. [コマンドプロンプト] を [管理者として実行] にて開始します。

1-2. 以下のコマンドを実行し、日本語の言語パックをアンインストールします。

lpksetup /u ja-JP

1-3. コンピューターを再起動します。

1-4. 日本語のユーザーインターフェイスを使用していたアカウントでサインインし、英語で表示されていることを確認します。

※ lpksetup コマンドをオプションなしで起動すると、GUI からのアンインストールが可能です。

※ 日本語が優先言語となっている場合、アンインストール実行後も設定アプリなどに残存する場合がありますが、Preferred languages 設定で "Japanese" を下に移動させて [Remove] を押すことで、アンインストールすることが可能です。

2.日本語の言語パッケージの再追加

-----------------------------------

日本語の言語パッケージの再追加と更新プログラムの再適用を行います。

なお、言語パッケージの再追加方法は、Windows Update エンドポイント宛の通信可否によって手順が変わるため、それぞれの手順について記載します。

<A.Windows Update エンドポイント宛の通信可能な環境での再追加手順>

2-1.弊社カタログサイトより、現在適用されている更新プログラムと同じバージョンの msu ファイルをダウンロードしておきます。

　Microsoft Update Catalog

　 https://www.catalog.update.microsoft.com/home.aspx

参考）現在適用されている更新プログラムの確認方法

　[ファイル名を指定して実行] で "winver" を実行して表示される [Windows のバージョン情報] ウインドウにて現在の OS ビルドの番号を確認します。

　次にWindows Server 2022 の更新履歴から同じビルド番号の更新プログラムを特定します。

　Windows Server 2022 更新履歴 - Microsoft サポート

　https://support.microsoft.com/ja-jp/topic/windows-server-2022-%E6%9B%B4%E6%96%B0%E5%B1%A5%E6%AD%B4-e1caa597-00c5-4ab9-9f3e-8212fe80b2ee

2-2. 下記弊社ブログの「オンラインでインストールが可能な場合」の項を参考に日本語の言語パッケージを再追加します。

　Windows Server 2022 の日本語化

　https://jpwinsup.github.io/blog/2025/12/14/Performance/LangPack/WindowsServerJapanize/

2-3. 言語パッケージ追加後、[コマンドプロンプト] を [管理者として実行] にて開始します。

2-4. 以下の一連のコマンドを実行し、作業フォルダー作成の上、事前にダウンロードしておいた msu ファイルの展開 (2 回) を行います。

2-4-1. 作業フォルダー作成

構文)

md <フォルダー (1 つ目)>

md <フォルダー (2 つ目)>

実行例)

md C:\KBTemp\cab

md C:\KBTemp\cab\cab2

2-4-2. 1 回目の展開

構文)

expand -f:Windows*.cab <msu ファイルの場所> <抽出先 (1 つ目のフォルダー)>

実行例)

expand -f:Windows*.cab C:\windows10.0-kb5082142-x64_0d2677e2b8f6550a978bd89e8991c4d3813ccf1f.msu C:\KBTemp\cab

2-4-3. 2 回目の展開

構文)

expand -f:* <1 回目で展開された cab> <抽出先 (2 つ目のフォルダー)>

実行例)

expand -f:* C:\KBTemp\cab\Windows10.0-KB5082142-x64.cab C:\KBTemp\cab\cab2

2-5.以下のコマンドを実行し、展開された cab ファイルを使って更新プログラムの再適用を行います。

構文)

DISM /Online /Add-Package /PackagePath:<2 つ目の作業フォルダーに展開された cab ファイル (※ KB 番号を冠するもの)>

実行例)

DISM /Online /Add-Package /PackagePath:C:\KBTemp\cab\cab2\Windows10.0-KB5082142-x64.cab

2-6. 言語パッケージ追加後、改めて更新プログラムの適用を行い、事象が解消することを確認します。

<B.Windows Update エンドポイント宛の通信ができない環境での再追加手順>

2-1.弊社カタログサイトより、現在適用されている更新プログラムと同じバージョンの msu ファイルをダウンロードしておきます。

　Microsoft Update Catalog

　 https://www.catalog.update.microsoft.com/home.aspx

参考）現在適用されている更新プログラムの確認方法

　[ファイル名を指定して実行] で "winver" を実行して表示される [Windows のバージョン情報] ウインドウにて現在の OS ビルドの番号を確認します。

　次にWindows Server 2022 の更新履歴から同じビルド番号の更新プログラムを特定します。

　Windows Server 2022 更新履歴 - Microsoft サポート

　https://support.microsoft.com/ja-jp/topic/windows-server-2022-%E6%9B%B4%E6%96%B0%E5%B1%A5%E6%AD%B4-e1caa597-00c5-4ab9-9f3e-8212fe80b2ee

2-2.下記弊社ブログの「オンラインでインストールできない場合」の項を参考に日本語の言語パッケージを再追加します。

　Windows Server 2022 の日本語化

　https://jpwinsup.github.io/blog/2025/12/14/Performance/LangPack/WindowsServerJapanize/

2-3. 言語パッケージ追加後、[コマンドプロンプト] を [管理者として実行] にて開始します。

2-4. 以下の一連のコマンドを実行し、作業フォルダー作成の上、事前にダウンロードしておいた msu ファイルの展開 (2 回) を行います。

2-4-1. 作業フォルダー作成

構文)

md <フォルダー (1 つ目)>

md <フォルダー (2 つ目)>

実行例)

md C:\KBTemp\cab

md C:\KBTemp\cab\cab2

2-4-2. 1 回目の展開

構文)

expand -f:Windows*.cab <msu ファイルの場所> <抽出先 (1 つ目のフォルダー)>

実行例)

expand -f:Windows*.cab C:\windows10.0-kb5082142-x64_0d2677e2b8f6550a978bd89e8991c4d3813ccf1f.msu C:\KBTemp\cab

2-4-3. 2 回目の展開

構文)

expand -f:* <1 回目で展開された cab> <抽出先 (2 つ目のフォルダー)>

実行例)

expand -f:* C:\KBTemp\cab\Windows10.0-KB5082142-x64.cab C:\KBTemp\cab\cab2

2-5.以下のコマンドを実行し、展開された cab ファイルを使って更新プログラムの再適用を行います。

構文)

DISM /Online /Add-Package /PackagePath:<2 つ目の作業フォルダーに展開された cab ファイル (※ KB 番号を冠するもの)>

実行例)

DISM /Online /Add-Package /PackagePath:C:\KBTemp\cab\cab2\Windows10.0-KB5082142-x64.cab

2-6. 言語パッケージ追加後、改めて更新プログラムの適用を行い、事象が解消することを確認します。

English

Hello, this is the SQL Server Support Team.

In this article, we explain an issue where operating system updates (such as Windows Updates released in and after May 2026) may fail with error 0x800f081f or 0x800f0831 after localizing an Azure Marketplace image containing SQL Server (for example, SQL2022-WS2022) to Japanese.

Issue

Installation of Windows updates may fail with a message similar to the following:

"更新プログラムのインストール中に問題が発生しましたが、後で自動的に再試行されます。この問題が引き続き発生し、 Web 検索やサポートへの問い合わせを通じて情報を集める必要がある場合は、次のエラーコードが役立つ可能性があります。: (0x800f081f)."

Note: In some cases, the reported error code may be 0x800f0831 instead.

Cause

This issue may occur due to a problem in certain Marketplace images that include SQL Server.

The issue has already been fixed in the latest Marketplace images. Therefore, it does not affect newly deployed servers created from the updated images.

How to Identify the Issue

You can determine whether your environment is affected if the following warning appears in C:\Windows\Logs\CBS\CBS.log, indicating that the manifest file below cannot be found:

amd64_microsoft-hyper-v-ram-parser.resources_31bf3856ad364e35_10.0.20348.1_ja-jp_531076835e0e2f98.manifest

File: C:\Windows\Logs\CBS\CBS.log

2026-05-19 12:11:57, Info CBS Exec: Warning - Manifest doesn't exist for: \\?\C:\Windows\CbsTemp\31254331_3251161130\Windows10.0-KB5087545-x64.cab\amd64_microsoft-hyper-v-ram-parser.resources_31bf3856ad364e35_10.0.20348.1_ja-jp_531076835e0e2f98.manifest

Resolution

Important Notes

Before proceeding, please ensure the following:

Create a backup of the Azure VM using Azure Backup so that the VM can be restored to its pre-maintenance state if necessary.
Stop the SQL Server service before performing the procedure, and set its startup type to Disabled.
After the procedure is completed, restore the SQL Server service startup type to its original setting (such as Automatic or Automatic (Delayed Start)).
Do not restart the operating system after reinstalling the Japanese language pack until the update has been reapplied successfully.
Immediately after the language resources are reinstalled, the versions of the OS binaries and language resources may not match. Restarting at this stage can cause issues such as the inability to establish Remote Desktop connections.

1.Remove the Japanese Language Pack

1-1. Launch Command Prompt as Administrator.

1-2. Run the following command to uninstall the Japanese language pack:

lpksetup /u ja-JP

1-3. Restart the computer.

1-4. Sign in using an account that previously used the Japanese user interface and confirm that Windows is now displayed in English.

Notes

If Japanese remains listed because it is configured as the preferred language, move Japanese lower in the Preferred languages list and select Remove.

2.Reinstall the Japanese Language Pack

Reinstall the Japanese language pack and then reapply the Windows update.

The procedure differs depending on whether the server can communicate with Windows Update endpoints.

<Environment with Access to Windows Update Endpoints>

2-1. Download the Corresponding MSU File

Download the .msu file that matches the update version currently installed on the server from the Microsoft Update Catalog.

Microsoft Update Catalog

https://www.catalog.update.microsoft.com/home.aspx

How to Identify the Installed Update Version

Run winver.
Note the current OS build number shown in the About Windows dialog.
Identify the corresponding update from the Windows Server 2022 update history page.

Windows Server 2022 Update History

https://support.microsoft.com/en-us/topic/windows-server-2022-update-history-e1caa597-00c5-4ab9-9f3e-8212fe80b2ee

2-2. Reinstall the Japanese Language Pack (Online)

Perform the following steps:

2-2-1. Go to Settings > Time & Language > Language & Region, then click Add a language.

2-2-2. On the Choose a language to install screen, select 日本語 (Japanese) and click Next.

2-2-3. On the Install language features screen, select Language pack and click Install.

2-2-4. Wait for the installation to complete.

2-2-5. Once the installation is complete, sign out and then sign back in. After that, you will be able to select Japanese as your display language under [Preferred languages].

2-3. Launch Command Prompt as Administrator.

2-4. Extract the Downloaded MSU File

2-4-1. Create Working Directories

md <Folder1>

md <Folder2>

2-4-2. First Extraction

expand -f:Windows*.cab <MSU file> <Folder1>

2-4-3. Second Extraction

expand -f:* <CAB extracted in previous step> <Folder2>

2-5. Reapply the Update

Use DISM to reinstall the extracted update package.

DISM /Online /Add-Package /PackagePath:<Folder2>\<KB CAB file>

2-6. After the language pack has been reinstalled, install the latest update again and verify that the issue is resolved.

<Environment Without Access to Windows Update Endpoints>

2-1. Download the Corresponding MSU File

Download the .msu file matching the currently installed update from the Microsoft Update Catalog.

Microsoft Update Catalog

https://www.catalog.update.microsoft.com/home.aspx

Refer to the procedure described above to identify the correct update package.

2-2. Reinstall the Japanese Language Pack (Offline)

Perform the following steps:

2-2-1. Refer to the link below and download the required media according to your licensing agreement.
For Windows Server 2022, you will need the "Languages and Optional Features for Windows Server 2022" media.

https://learn.microsoft.com/en-us/microsoft-365/commerce/licenses/download-vl-products?view=o365-worldwide&viewFallbackFrom=o365-wo%25E2%2580%25A6

2-2-2. Mount the downloaded ISO file to the target virtual machine.

2-2-3. Right-click the Start menu and select Run. Then enter lpksetup.exe and press Enter.

2-2-4. Click Install display languages. On the next screen, click Browse and select the drive that was created when you mounted the ISO file in Step.

2-2-5. Select Microsoft-Windows-Server-Language-Pack_x64_ja-jp and click OK.

2-2-6. Verify that Japanese (日本語) is selected, and then click Next.

2-2-7. Review the license terms. If you agree, select I accept the license terms, and then click Next to begin the installation.

2-2-8. When the installation is complete and Completed is displayed, click Close.

2-3. Launch Command Prompt as Administrator.

2-4. Extract the Downloaded MSU File

2-4-1. Create Working Directories

md <Folder1>

md <Folder2>

2-4-2. First Extraction

expand -f:Windows*.cab <MSU file> <Folder1>

2-4-3. Second Extraction

expand -f:* <CAB extracted in previous step> <Folder2>

2-5. Reapply the Update

Use DISM to reinstall the extracted update package.

DISM /Online /Add-Package /PackagePath:<Folder2>\<KB CAB file>

2-6. After the language pack has been reinstalled, install the latest update again and verify that the issue is resolved.

Thanks.

Cost Management with Azure Resource Manager MCP

demiajayi — Wed, 26 Aug 2026 02:17:28 GMT

Today, we’re announcing new Cost Management capabilities in the Azure Resource Management (ARM) MCP server. A core set of cost tools is now available by default, making it easier for AI agents to bring cost context into Azure workflows without additional configuration. For more advanced scenarios, you can enable the optional Cost Management toolset to access an expanded set of capabilities.

Cloud operations and cloud economics are deeply connected, but the data required to make those decisions is often fragmented across different tools and experiences. The default cost capabilities help AI agents combine Azure resource information with essential cost insights in the workflows where cloud decisions are made. When you need broader coverage, the optional Cost Management toolset extends that experience with additional tools for deeper analysis, planning, and optimization.

What You Can Do Today

Plan Before You Deploy

Every cloud project starts with a series of tradeoffs. Before resources are deployed, teams need to understand how architectural decisions, resource choices, and regions will impact cost.

An AI agent can estimate costs, compare deployment options, and factor in negotiated rates where available so you can evaluate cost alongside technical requirements before deployment.

Examples

"Estimate the monthly cost of deploying a three-node AKS cluster with my negotiated pricing.”
“How would my estimated monthly cost change if I updated the VM in this ARM template <ARM Template> from Standard_D8s_v5 to Standard_D2s_v5? and is that within our organization’s policy?”

Understand Costs

Once workloads are running, cost visibility becomes the priority. An AI agent can query your cost data to explain cost trends, changes in spend, and surface cost drivers.

Example

"What were my top cost drivers this month?"

Manage Cloud Spend

With visibility in place, budget and alerts help teams avoid surprises. An AI agent can create and review budgets and check alerts so you can see where spend is tracking against your plans.

Example

“How am I doing against my budgets this month?”

Understand and Manage Spend

As cloud environments grow, identifying the highest-impact savings opportunities becomes increasingly challenging.
An AI agent can surface appropriate Advisor cost recommendations so you can prioritize actions to improve savings.

Example

"Show me any eligible Savings Plans recommendations that would save me at least $200/month.”

Analyze AKS Workloads

If you’re running AKS workloads on Azure, an AI agent can query AKS cost data by cluster and namespace and compare active and idle capacity.

Example

“Show me how much of my AKS spend is idle vs actively used in my current subscription”

Make Cost-Aware Azure Decisions

Whether you're evaluating a new deployment, investigating spending changes, tracking budgets, identifying savings opportunities, or analyzing AKS workloads, the Cost Management capabilities in the Azure Resource Management (ARM) MCP server help bring financial context directly into Azure workflows. By combining Azure resource information with cost, pricing, budget, and optimization insights, AI agents can help you make more informed decisions without switching between multiple tools and experiences.

What's Included

The Azure Resource Management (ARM) MCP server supports the following Cost Management scenarios:

Query – Explore historical spend, understand cost trends
Forecast – Project future costs based on current usage
Budgets and alerts – Create, review, and monitor budgets and alerts to help teams stay ahead of overruns.
Reservations and Savings Plans – Review recommendations and utilization data to identify commitment-based savings opportunities.
Pricing – Retrieve retail pricing and negotiated price sheet details where available.

Default Cost Management capabilities

A core set of tools is enabled by default, allowing agents to support common cost analysis and pricing scenarios without additional configuration.

Capabilities	Supported Tools
Query	query_costs query_aks_costs
Pricing	get_retail_prices start_pricesheet_download get_pricesheet_status

Additional Cost Management capabilities

For broader cost-management scenarios, you can enable the optional Cost Management toolset. This extends the default experience with additional tools for dimensions, forecasting, budgets, alerts, reservations, and Savings Plans.

Capabilities	Tools
Query	list_dimensions
Forecast	forecast_costs
Budget & Alerts	list_budgets get_budget create_budget list_alerts
Reservations & Savings Plans	get_benefit_recommendations list_benefit_utilization list_reservation_transactions

Getting Started

Pre-Requisites

VS Code installed
Valid Azure account with appropriate permissions
GitHub Copilot subscription

Installation

Install ARM MCP - https://aka.ms/JoinARMMCP
- VS Code launches automatically
- Click Install under Azure Resource Manager MCP Server
- Sign in with your Azure credentials
- If you hit any authentication issues see the Troubleshooting Guide in the ARM MCP repo

Add cost header to the Azure Resource Manager MCP configuration file for access to optional cost tools
- Opt-in per client with x-mcp-toolset header: CostManagement
- For GitHub Copilot Chat (in VS Code)
  - Configure mcp configuration file for a specific workspace or globally
    - Option 1 (Workspace): configure vscode/mcp.json file in your repo
    - Option 2 (User): Edit mcp.json file in User Configuration:
  - Open the Command Palette (Ctrl+Shift+P / Cmd+Shift+P)
  - run MCP: Open User Configuration and edit the mcp.json it opens

{ "servers": { "azure-resource-manager-mcp": { "type": "http", "url": "https://mcp.management.azure.com", "headers": { "x-mcp-toolset": "CostManagement" } } } }

- - Save the file, then restart the server: open the Command Palette and run MCP: List Servers, pick azure-resource-manager-mcp, and choose Restart. The new tools will appear under Configure Tools in Chat.

- For GitHub Copilot CLI
  - Edit ~/.copilot/mcp-config.json with the relevant CostManagement header

{ "mcpServers": { "azure-resource-manager-mcp": { "type": "http", "url": "https://mcp.management.azure.com", "headers": { "x-mcp-toolset": "CostManagement" } } } }

- - Restart the MCP server to pick up the change. In Copilot CLI today this means exiting and relaunching copilot. Verify the server is running with the /mcp show slash command inside copilot.

What's Next?

Microsoft Cost Management is actively expanding our agentic cost capabilities. Upcoming capabilities will include additional supported tools and skills.

Give Feedback

We want to hear from you. Please share your feedback here - https://aka.ms/azurecost/mcptools/give/feedback

Resources

- 📖 Documentation – Complete setup and usage guide

- 🔗 Feedback – Share your feedback on Cost tools in ARM MCP

- 🐛 Report Issues - Report bugs with ARM MCP

- 🛠️ Troubleshooting – Resolve common issues with ARM MCP

Maia 200: Software-defined dataflow and all-Ethernet networking for efficient inference on Azure

SherryX19 — Wed, 26 Aug 2026 14:40:52 GMT

By Sherry Xu, Prashant Ranjan, Torsten Hoefler

As we enter the era of frontier-scale intelligence, the economics of AI are increasingly determined not by peak compute performance but by how efficiently an entire system can generate tokens at scale. Production inference workloads are rapidly expanding beyond interactive chat toward an array of Copilots, coding assistants, deep reasoning, multimodal experiences, agentic workloads, and synthetic data generation. Azure Maia 200, Microsoft’s second-generation AI accelerator, is our latest custom silicon, engineered from the ground up for efficient inference at cloud scale.

At approximately 30% better performance per dollar than the latest-generation GPUs in Microsoft’s fleet, Maia 200 is built on one central principle: that co-optimizing the models, application harnesses, reinforcement learning environments, kernels, communication library, systems, and custom silicon will lead to the best customer outcomes – per token, per watt, and per dollar.

There is no single specification that can achieve this, and no equation that drives faster math. System-level efficiency emerges from co-optimizing models, software, data placement, communication, and silicon together. At Hot Chips 2026, we are pleased to share the foundation behind that effort: the Software-defined Local Access architecture embodied in Maia 200.

Inference is becoming a systems challenge

Inference is no longer a compute problem alone—it is a systems problem that spans memory, networking, and software. As the range of production inference use cases continues to expand, workloads have also become more diverse and demanding.

Different phases of inference place different stresses on infrastructure: processing an initial prompt often requires substantial compute, while generating tokens over time can become constrained by memory bandwidth and communication. Emerging model architectures, including Mixture-of-Experts designs, sparse execution techniques, and advanced KV-cache strategies, further increase the importance of efficiently moving and managing data. At the same time, the largest models now span many accelerators, making communication between devices a critical factor in overall performance.

Meeting these demands requires optimizing for more than peak operations per second. An effective inference platform must deliver high throughput, low latency, reliability, and model quality while minimizing cost and energy per token. Maia 200 was designed with this broader objective in mind, and approaches this challenge using the architectural principles of Software-defined Local Access—both inside the accelerator and scaling to an all-Ethernet architecture that extends explicit memory orchestration across the system.

Software-defined Local Access: making movement explicit

Software-defined Local Access, or SDLA, is a dataflow architecture that gives software direct control over how data moves between high-bandwidth memory and localized, highly specialized SRAMs. Rather than relying on implicit cache behavior, SDLA separates control from data movement and gives software direct control over both memory movement and placement using Direct Memory Access (DMA) engines, compute engines, network operations, and synchronization. This approach helps the system achieve more predictable execution characteristics, enabling more consistent performance between runs.

Maia 200 brings the SDLA principles together in a vertically co-designed system-on-chip. Its control, compute, memory, I/O, and synchronization structures are designed to operate concurrently rather than compete for a single control path. The purpose of the architecture is to keep the compute engines productively fed by placing, reshaping, moving, and synchronizing data efficiently.

All-Ethernet networking: extending SDLA beyond the chip

As models span accelerators, networking becomes part of the execution engine. Maia 200’s two-tier scale-up network applies an all-Ethernet approach, combining a standards-based physical and switching foundation with AI-specific endpoint hardware, transport, topology, congestion management, reliability mechanisms, and collective software.

Maia 200 delivers scalable, predictable performance for large AI inference deployments through an advanced two-tier, Ethernet-based scale-up networking architecture. Intelligent load balancing distributes traffic across on-chip resources, while built-in resilience mechanisms quickly adapt to network disruptions. This enables customers to run dense inference workloads more efficiently, maximize cluster utilization, reduce stranded capacity, and lower overall infrastructure costs.

We have contributed this direction to the Ultra Ethernet Consortium (UEC)’s AI base transport profile, helping establish Ethernet as a common foundation for networking across the industry.

Figure 1. Maia 200 uses a unified Ethernet hierarchy based on a variant of the HammingMesh topology: four accelerators form a directly connected quad, 48 accelerators form a rack-scale domain, and a two-tier switched network can scale to 6,144 accelerators.

Sustaining performance across real AI workloads

The value of the SDLA architecture appears in sustained kernel performance rather than peak specifications alone. To evaluate Maia 200 under realistic conditions, we measured performance across 6,143 matrix-multiplication shapes representative of production inference workloads. Across these scenarios, Maia 200 kept compute engines highly utilized, minimizing idle time by coordinating computation, memory, and communication as a single workload. Even as AI workloads shift between heavy computation (e.g., prompt processing), memory access (e.g., token generation), and communication (e.g., collective operations such as Allgather), Maia 200 maintains consistently predictable performance—treating the accelerator, memory, network, and software as one coordinated system.

Figure 2. FP8 matrix-multiplication performance across the same set of inference-relevant shapes, showing high utilization across compute- and memory-bound operating points.

Figure 3. Allgather performance on eight Maia 200 accelerators. Direct exchange minimizes latency for small transfers; ring exchange approaches the network bandwidth limit for larger transfers.

From silicon to useful AI systems

Maia 200 demonstrates Microsoft’s system co-design philosophy for model performance and efficiency. Maia 200 delivers over 40% higher token generation under the same rack power budget throughput when running the MAI-Thinking-1 model than other leading accelerators in the Azure fleet. This efficiency is only possible by co-designing models, application frameworks, reinforcement-learning environments, kernels, communication libraries, systems software, and custom silicon as a unified stack. At the same time, Maia 200 maintains the flexibility to support a broad ecosystem of open models, enabling customers to optimize for both performance and choice.

As Microsoft continues along a path of co-designing models and infrastructure, Maia 200 provides a foundation for improving quality, performance, cost, and energy efficiency for agentic inference workloads. We look forward to advancing these SDLA architectural principles in future generations of Maia system, and invite you to see the full details and data on Maia 200 operations in our latest paper on arXiv:[2608.24664] Maia 200: A Software Defined Dataflow System for Large-scale AI Acceleration.

To learn more, explore our blogs on Maia 200:

Instant revocation of service principal bearer tokens with CAE

Simone_Oor — Wed, 26 Aug 2026 14:48:29 GMT

Introduction

Workload identities such as service principals are an important part of the attack surface in Entra: they are often used by critical automated processes, and they can carry privileged and sensitive permissions. The authentication flow used by these principals is the Client Credential flow, which results in an access token.

In this blog post, I'll describe how Continuous Access Evaluation (CAE) can be leveraged to instantly invalidate an access token for a service principal in Microsoft Entra. This is incredibly valuable as part of a kill-switch in incident response scenarios.

Access tokens (aka bearer tokens)

Access tokens are bearer tokens. Instant invalidation of bearer tokens is an important response capability because normally these tokens have a lifetime between 60 and 90 minutes.

This is a long time in scenarios where a compromised token can be used for triggering malicious automation by a bad actor.

How to enable CAE for service principal tokens

CAE is enabled by a “client capability” claim, “xms_cc”, with the value of “cp1”.

The screenshot below shows an example of a PowerShell function that incorporates this:

Recognizing a CAE-enabled token

The access token I get back from this request is base 64 encoded. I can echo this base 54 encoded value back in my PowerShell session, then paste it into a decoder to investigate it (jwt.io or jwt.ms).

The first thing you will notice is the presence of this claim:

The second thing you will notice is the lifetime of the token:

The expiry value is now (roughly) 24 hours after the “iat” (issued at) time, instead of the 60-90 minutes that we normally see in tokens that are not CAE-enabled.

Why does this matter

Although a longer lifetime seems less secure, Microsoft’s philosophy around this is that with Workload Identity Protection enabled, risk levels can dynamically be picked up for the identity.

If an identity is at a high risk level, the token is revoked within a couple of minutes automatically.

High risk is only one of three revocation events. At the time of writing, the revocation events are:

Service Principal at high risk
Service Principal disabled
Service Principal deleted

How to manually test this

In PowerShell, I use the “Connect-And-GetToken" function above to obtain the token and construct a header, which I then use to call a Graph endpoint:

(I can call this https://graph.microsoft.com/v1.0/users endpoint, because my app, represented by $cid, has the User.Read.All API permission. This is not relevant for this topic, just an example for demo purposes).

Whilst the bearer token (in the header) is still valid, the response that comes back can be something like this:

If this were not a CAE-enabled token, even removing the service principal, changing its password or disabling it, would NOT stop me from doing this, as long as the bearer token lifetime has not expired!

Now let’s trigger revocation manually.

I will describe two of the three revocation events: high risk and disablement.

Test 1: Simulate service principal at high risk

To simulate putting the service principal at risk level high, we use another service principal that has the API permission “IdentityRiskyServicePrincipal.ReadWrite.All”.

With this other identity (connect-MgGraph), issue this command:

Confirm-MgRiskyServicePrincipalCompromised -ServicePrincipalIds @("{Object id of our target service principal}”)

After a short while, the identity shows as high risk in the portal (under Identity Protection > Risk workload identities).

Now, when I try to use the same header with the same access token from before, the response I get is:

“Invoke-RestMethod : The remote server returned an error: (401) Unauthorized.”

This means that the access token was effectively revoked and the revocation objective was achieved.

To dismiss (cancel) the risk status on that service principal, click on “Dismiss service principal(s) risk” in the portal (top bar), or use Invoke-MgDismissRiskyServicePrincipal in PowerShell.

After this risk dismissal, my token and the header containing it are still invalid, so I have to re-authenticate (in my case with the “Connect-And-GetToken” function) if I again want to make the call to Graph.

‼️This brings me to an important point: this option must be paired with Conditional Access for Workload Identities, using a policy that BLOCKS authentication in case the risk level is high.‼️

Without this, my token may be revoked by the risk event, but a bad actor that also owns the client ID and secret could, in theory, re-authenticate and get a new token that subsequently does not get revoked as the revocation event on the previous token is in the past (unless a new risk event is picked up).

Conditional Access for workload identities requires a Workload Identity Premium license.

Test 2: Simulate service principal disabled

A simpler and faster option for revoking a CAE-enabled service principal access token is to disable the service principal.

For this, I use the Graph Explorer in my lab:

‼️Notice that this must be done directly on the service principal. Deactivating the registered app also disables the service principal, but it does not trigger the revocation event.‼️

This instantly gives me a “(401) Unauthorized” if I try to reuse the header (with the token), and therefore it is a better manual kill switch option.

I was able to do this in a tenant without a Workload Identity Premium license.

Microsoft Entra Sign-in logs

Service principals have their own section in the Microsoft Entra Sign-in log.

If the token issued (after the sign-in was successful) was CAE enabled, the “Basic in” tab of the sign-in event will show a line "Continuous access evaluation Yes",

Under “Additional details”, you will see:

Warning

The risk level may be reaching "high" without your input, which leads to automatic revocation for CAE-enabled tokens.

Detections at the time of writing include:

Microsoft Entra threat intelligence
Suspicious Sign-ins
Admin confirmed service principal compromised (as we have done in this blog)
Leaked Credentials
Malicious application
Suspicious application
Anomalous service principal activity
Suspicious API Traffic

Securing workload identities with Microsoft Entra ID Protection - Microsoft Entra ID Protection | Microsoft Learn

Conclusion

To create the ability to revoke access tokens before the end of their lifetime, enable the "cp1" client capability claim at integration time (when building your solutions and their authentication call).

You cannot revoke service principal access tokens that were not created with the client capabilities claim of “cp1” in the first place.

Please refer to Continuous access evaluation for workload identities in Microsoft Entra ID - Microsoft Entra ID | Microsoft Learn for an up-to-date list of possibilities and limitations for this feature.

I am currently researching if this can be used in Agent flows, where the token exchange pattern applies, so stay tuned.

Learn more

Access token claims reference - Microsoft identity platform | Microsoft Learn

Microsoft Entra Conditional Access for workload identities - Microsoft Entra ID | Microsoft Learn

Continuous access evaluation for workload identities in Microsoft Entra ID - Microsoft Entra ID | Microsoft Learn

Graph Explorer | Try Microsoft Graph APIs - Microsoft Graph

Securing workload identities with Microsoft Entra ID Protection - Microsoft Entra ID Protection | Microsoft Learn

Retirement of Microsoft HPC Pack

XinXin — Wed, 26 Aug 2026 16:54:57 GMT

Overview

Microsoft HPC Pack is a Windows‑based high‑performance computing (HPC) scheduler that enables customers to deploy, manage, and operate HPC workloads in on‑premises and hybrid environments. Since its original release, HPC Pack has supported a range of Windows‑centric HPC scenarios, including job scheduling, workload orchestration, and integration with Windows Server–based technologies.

Microsoft is announcing the planned retirement of HPC Pack. For customers looking to run HPC workloads going forward, Microsoft’s supported platform for managed HPC and parallel workloads is Azure Batch. Customers may also adopt any other Azure service as appropriate for their workload. These platforms provide modern, cloud‑native capabilities for scheduling, scaling, and operating HPC workloads on Azure.

Important Dates

Retirement announcement: August 27, 2026
End of Support / retirement date: August 27, 2027

After August 27, 2027, HPC Pack will no longer receive feature updates, bug fixes, or standard product support. No new versions or enhancements will be released. Existing HPC Pack deployments will not be forcibly disabled; however, they will be considered unsupported.

Support During the Retirement Period

HPC Pack is now entering a one-year retirement period that ends at End of Support on August 27, 2027. Microsoft will provide limited, retirement-only support during this period. The scope of this support is defined below so customers know what to expect.

What Microsoft will provide during the retirement period:

Security response, including investigation of applicable security vulnerabilities (CVEs) and any security updates deemed necessary by Microsoft
Guidance based on existing Microsoft documentation, published best practices, and previously validated HPC Pack configurations
Assistance with support incidents involving supported HPC Pack components and documented product functionality

What is NOT included during the retirement period:

Non-security bug fixes
Performance, reliability, scalability, or optimization improvements
New features or feature enhancements
Customer-specific hotfixes, custom code changes, or design modifications
Validation or certification of new operating systems, hardware platforms, drivers, firmware, third-party software, or dependencies
Validation of new deployment architectures, configurations, or integration scenarios
Changes to existing product behavior or design
Creation of new documentation, guidance, or troubleshooting content
Troubleshooting that requires new product investigation beyond existing product knowledge, documentation, or previously validated scenarios
New customer onboarding, solution design, or pre-sales assistance
Migration planning, migration execution, or migration consulting services
Support for configurations, integrations, or deployment scenarios that are outside published documentation, established best practices, or previously validated HPC Pack environments

After End of Support (August 27, 2027): HPC Pack will receive no further updates, fixes, or technical support, and deployments will be considered unsupported.

What Customers Should Do

Customers currently using HPC Pack should begin planning migration to a supported alternative as soon as possible. For most scenarios, the following services are recommended:

Azure Batch – for managed scheduling and execution of parallel and HPC workloads
Or any other Azure service as appropriate for your workload

Resources

HPC Pack documentation: Microsoft HPC Pack 2019
Public issue tracker: Azure/hpcpack GitHub repository

Next Steps

Customers who anticipate needing additional planning assistance or migration guidance should engage early with their Microsoft account team or Cloud Solution Architect (CSA) to help ensure a smooth transition and avoid potential disruption. To help us understand customer needs and improve future communications, please complete this Microsoft Form. Share any technical, operational, or business challenges you anticipate during your transition. Your feedback will help us identify common concerns and refine future guidance.

Tracking Batch node state and duration in Log Analytics

Ahmed_Khaled — Tue, 25 Aug 2026 18:04:45 GMT

1. The problem (a real customer scenario)

Some of our customers described that they have those underlying gaps:

Symptom	What the customer wanted
Nodes intermittently stayed in the leavingpool state for hours. The customer needed to know which node was stuck and for how long, so an alert could fire only after a threshold (e.g. > 60 minutes).	A dashboard/query that isolates one node at a time and calculates its duration in the current state.
A node was reported unusable because the underlying OS provisioning failed. The customer needed automated detection to remediate (drop the pool to 0 and back up) instead of catching it manually.	A per-node feed that carries state + error code + IP, so an alert can identify the offending node and its error.

In both scenarios we recommended the same workaround which was: Build a small app that polls the Batch REST API for (per-node data) and stores it.

In this Article we will build this implementation as a reusable component and show the query and alert that would solve similar problems.

2. Why the platform cannot show this directly

- The main issue came from the fact that Azure Monitor metrics tools don't provide granular, node-level tracking out of the box (no 'NodeId' or 'PoolId'). Instead, it has a rich metrics that can help monitoring the Batch accounts (see the full metrics).

- All those metrics are single integers for the whole account. The "Split-by" filter in the portal Metrics blade lists no options for these two metrics so the customers can't filter down into a specific node's status.

- While AzureDiagnostic logs successfully show events at the pool level (example: 'PoolResizeCompleteEvent' or 'PoolResizeCompleteEvent'), they don't show state-change events for individual nodes. The current schema simply doesn't contain a 'NodeStateChange' row, meaning old KQL queries looking for this event will return zero results.

In summary, using the current Azure monitor metrics can tell you that "3 nodes are leaving," which answers your "What" question.

But if you are asking "Who" are leaving, you won't get a direct answer.

The good news that this trivial question will be answered in that article ✌️

3. The solution

The solution is mainly divided in 2 parts:

1- What we can expose from Azure and build on top of:

As mentioned before, the Azure Batch REST API returns rich per-node data via GET /pools/{poolId}/nodes (check the sample response). The most relevant fields are: (state, schedulingState, stateTransitionTime, errors[], startTaskInfo, ipAddress, allocationTime, lastBootTime) that we will use them in the collector (2nd part of the solution).

2- A small polling collector

This is a python script runs on a schedule.

1- It calls List pools + List Nodes, and posts each snapshot as a row in the Log Analytics custom table "BatchNodeInventory_CL" which is retained by Log Analytics Workspace "LAW" standard retention.

2- It authenticates against the Batch service. In a production environment, this is securely handled using a Managed Identity (MI), meaning no hardcoded passwords or keys are required.

But in my repro, it simply uses a token generated via the CLI command:

az account get-access-token

The core loop is:

token = aad_token("https://batch.core.windows.net/") for pool in GET /pools: for node in GET /pools/{pool.id}/nodes: row = { TimeGenerated, PoolId, NodeId, State, SchedulingState, StateTransitionTime, VmSize, IpAddress, IsDedicated, RunningTasks, TotalTasksRun, ErrorCount, ErrorCode, ErrorMessage, StartTaskState, StartTaskExitCode, AllocationTime, LastBootTime } rows.append(row) post_to_law(rows)

You can find the full script here: node-state-collector.py

5. The query that answers the customer's question

Now, our customers' questions became just a query 😉

In my repro, I kept the collector running for few minutes before I start to query:

BatchNodeInventory_CL | where TimeGenerated > ago(1h) | summarize LastSnapshot=max(TimeGenerated), State=any(State_s), Pool=any(PoolId_s), IP=any(IPAddress), Transition=any(StateTransitionTime_t), ErrCode=any(ErrorCode_s) by NodeId=NodeId_s | extend MinutesInState = datetime_diff("minute", LastSnapshot, Transition) | project Pool, NodeId, State, IP, MinutesInState, ErrorCode | order by MinutesInState descEach row is one physical node with its pool, current state, IP, the number of minutes it has been in that state, and (when applicable) the failure code.

For the article's 1st scenario, the query becomes:

| where State == "leavingpool" and MinutesInState > 60

For the article's 2nd scenario, it becomes:

| where State == "unusable"

Both are one-line filters on the same table. You can also filter based on any node state, See: Node state counts

6. The alert

I bet that you won't need a fixed notifications just telling you that the system was checking in (heartbeats). You will need an intelligent alert that would only fire when a specific threshold was crossed.

By leveraging the custom table (BatchNodeInventory_CL) I created, setting up this alert becomes straightforward using a Log Analytics scheduled query rule:

BatchNodeInventory_CL | where TimeGenerated > ago(10m) | summarize LastSnapshot=max(TimeGenerated), State=any(State_s), Pool=any(PoolId_s), IP=any(IPAddress), Transition=any(StateTransitionTime_t), ErrCode=any(ErrorCode_s), ErrMsg=any(ErrorMessage_s) by NodeId=NodeId_s | extend MinutesInState = datetime_diff("minute", LastSnapshot, Transition) | where State in ("unusable","leavingpool","starttaskfailed","offline") or MinutesInState > 60

The KQL query for the alert is very similar to the diagnostic query we used earlier, but with a few critical modifications designed for automated monitoring:

Shorter Time Window (ago(10m)) as this alert rule will run frequently.
We added ErrMsg=any(ErrorMessage_s) to the summarize block. This ensures that when the alert fires, the on-call engineer can read the actual error description without having to dig through logs immediately.
The final line instructs Azure Monitor to trigger only if a node enters a distinctly bad state (like unusable, starttaskfailed, or offline) OR a node has been stuck in its current state (such as leavingpool) for more than 60 minutes (MinutesInState > 60)

This rule is typically bound to an Action Group (which handles notifications via email, Microsoft Teams, or PagerDuty). The generated payload directly includes the NodeId, Pool, State, ErrorCode, and MinutesInState. This means the on-call engineer receives the exact identifier of the failing node and the cause of the failure in a single, actionable message.

7. Bonus

Because we have already done the heavy lifting of continuously ingesting detailed, per-node data into the BatchNodeInventory_CL table, that exact same dataset can do double duty.

Without running any additional polling scripts, this single table can act as a live inventory dashboard, providing aggregate counts per pool.

BatchNodeInventory_CL | where TimeGenerated > ago(30m) | summarize Nodes=dcount(NodeId_s), Idle=dcountif(NodeId_s, State_s=="idle"), Running=dcountif(NodeId_s, State_s=="running"), Unusable=dcountif(NodeId_s, State_s=="unusable"), StartTaskFailed=dcountif(NodeId_s, State_s=="starttaskfailed") by Pool=PoolId_sThe KQL results

8. Deploying in Production

To make this solution reliable, the Python script needs to run as a continuous background process.

I would say that you have 3 infrastructure options for deploying this collector script:

Azure Function (Timer trigger, Python): In my viewpoint, it's ideal because it is serverless, executes on a cron schedule and integrates seamlessly with Managed Identities and Key Vault + it's good cost.
Container App Job (Schedule trigger): I would choose this if the collector needs to run within a private Virtual Network (VNet) to reach a secured Batch account, or if you require a custom base image.
A small VM with cron/Scheduled Task: I would recommend if you already have a management VM and prefer to utilize the current infra not to provision new Azure resources.

Important Notes:

An Azure Monitor Log Analytics Function is not viable here, as this solution requires initiating outbound REST API calls outside of the Log Analytics environment.
The Log Analytics Shared Key should be stored securely in a Key Vault, or alternatively, the ingestion method should be migrated to Data Collection Endpoints (DCE) / Data Collection Rules (DCR) to eliminate the need for shared keys entirely.
As mentioned previously, the local az account get-access-token command should be replaced with the Azure SDK's Managed Identity token flow.
This solution doesn't replace built-in monitoring. Fleet-wide metrics, task events, and pool events are all still worth ingesting into the same LAW so a single dashboard covers everything. This collector just adds one extra dimension that some of our customers asked for.

New blog articles in Microsoft Community Hub

Partner Blog | Building the foundation for AI: Cloud, data, security, and AI skills for partners

The foundation for AI is broader than AI skills

Validate your Frontier Transformation expertise

Prepare for the launch of growth margins with API readiness resources

Understanding and Reclaiming Storage After DELETE in Azure Database for PostgreSQL

Why DELETE Does Not Immediately Shrink Storage and Azure Monitor Storage Graphs Continue to Show High Storage Consumption

Storage That Can Be Reclaimed

What Autovacuum Actually Does

PostgreSQL uses autovacuum to clean up obsolete row versions in the background. It also updates planner statistics and helps prevent transaction ID wraparound.

A regular vacuum operation:

Check whether Autovacuum is enabled:

A high count may mean that autovacuum is not keeping up, large batch operations are producing dead tuples quickly, or old transactions are preventing cleanup.

Learn more about Autovacuum PostgreSQL: Documentation: 18: 19.10. Vacuuming

Following options helps us to reclaim/reuse the storage

Option 1: Standard VACUUM

Option 2: VACUUM FULL

To return unused table space to the operating system, PostgreSQL provides:

Please refer VACUUM FULL - PostgreSQL wiki

Option 3: pg_repack

Option 4: Partitioning for Future Prevention

For historical data such as audit logs, telemetry, or event tables, partitioning is often the best long-term solution. Instead of deleting millions of old rows, you can remove an old partition.

This approach is faster, produces less bloat, and reduces the pressure on Autovacuum.

Azure storage metrics may remain high after rows are deleted and vacuumed.

For example:

This can happen for two separate reasons:

As a result, a reduction in table size does not necessarily reduce the storage provisioned for the Flexible Server. Once Azure storage has grown, it generally should not be assumed that it will automatically shrink.

Best Practices to Avoid Storage Surprises

Conclusion

Your on-call rotation has a new member: 10 production incidents, end to end, with Azure SRE Agent

Table of Contents

TL;DR

1. Why the "investigation" half is the real prize

2. What Azure SRE Agent actually does

The three primary use cases

The five extension primitives

Integrations you can assume

Skills vs. custom agents vs. knowledge files

3. Anatomy of an agent-run incident

4. Build the demo lab

4.1 Resource group and agent

4.2 Choose a permission level deliberately

4.3 Connect ServiceNow

4.4 Connect deployment correlation

4.5 Create the demo resources

4.6 One custom agent per domain

4.7 Response plans

5. Set your guardrails before your first incident

5.1 Run modes

5.2 What the product blocks for you

5.3 Hooks: the guardrail you write yourself

5.4 The recommended production policy

6. The ten use cases

Use case #1 — App Service: HTTP 500 after deployment

What happened

How Azure Monitor detected it

How the agent found the cause

Agent action table

Recovery note

Try it yourself

Guardrails

Use case #2 — AKS: pods in CrashLoopBackOff

What happened

How Azure Monitor detected it

How the agent found the cause

Agent action table

Recovery note

Try it yourself

Guardrails

Use case #3 — Azure SQL Database: CPU saturation and timeouts

What happened

How Azure Monitor detected it

How the agent found the cause

Agent action table

Recovery note

Try it yourself

Guardrails

Use case #4 — Azure Cosmos DB: HTTP 429 throttling

What happened

How Azure Monitor detected it