Azure topics

Cloud-Native vs. Hybrid for the 2026 Workplace

Gaaleh-Mem — Fri, 01 May 2026 14:54:03 GMT

When to choose Cloud-Native vs. Hybrid for the 2026 Workplace?

Hi everyone,

I am starting a discussion on the foundational phase of one project. As a Computer Engineer, I believe the most critical decision we face in 2026 is determining exactly when to step to a Full Cloud model versus maintaining a Hybrid Infrastructure.

In my view, the decision is not about cost, it is about resiliency, high availability and more avalability. I would like to exchange views with other engineers on these area: latency, edge requirements, integration and aglility.

In your experience, what are the Tipps that makes you choose one over the other for a 2026 environment?

I'm looking for technical architectural insights, not sales approaches.

Azure Automation Hybrid Runbook Worker Supported OS

PhilippZiemke — Thu, 30 Apr 2026 08:51:30 GMT

Hi everyone,

we are currently in the process of updating or environment to Server 2025. Since the mainstream support of Server 2022 ends October this year, we would also like to update our on-premise Azure Automation Hybrid Runbook Worker from 2022 to 2025.

As far as I can see from the documentation, OS is only supported up to Server 2022, but not Server 2025. Since the mainstream support end is closing in, is there any information on official support for Server 2025 for Azure Automation HRWs? Do you already have one successfully running with Server 2025?

Thanks!

Patterns for low-code Azure config state snapshot + recovery solution for resource groups

nicksal — Thu, 30 Apr 2026 02:08:17 GMT

I’m looking for patterns that capture resource configuration changes over time and support best-effort recovery (redeployment) of resource config state.

I understand that authoritative IaC (Bicep) would be the most mature option, however, I am wondering if anyone has ever implemented a solution similar to what I have described above.

Ideally this would be a low-code, Azure native solution.

Using Github Copilot from Azure Subscription

MSOPS1 — Wed, 29 Apr 2026 10:28:33 GMT

Hello,
I have a question on how GitHub Copilot can be accessed and managed through an Azure subscription. If I am getting a Github Copilot license, how is my azure subscription getting linked to the billing and licensing?
Specifically, I would like clarification on how the Azure subscription is linked to GitHub Copilot billing and licensing.

MFA required for Global Admin without Conditional Access or PIM enforcement

schiachris — Tue, 28 Apr 2026 15:06:56 GMT

Hi,

I'm analyzing a break-glass account scenario in Microsoft Entra ID and would like to validate a behavior I'm observing.

The account:

Has Global Administrator role (permanent assignment)
Is excluded from all Conditional Access policies (fully validated)
Is excluded from Authentication Methods policies and MFA Registration Campaign (fully validated)
Has no per-user MFA enabled (disabled)
PIM is not enforcing MFA (role is permanently active, no activation required)
Security Defaults are disabled
SSPR is not enforcing MFA

All configurable sources that could require MFA have been reviewed and fully ruled out.

However, when signing into Microsoft Admin Portals (Entra/Azure), MFA is still required and cannot be skipped.

In Sign-in logs:

Conditional Access → Not Applied
Authentication Details show:
"MFA required in Azure AD"
"App requires multifactor authentication"

Additionally, there is a Microsoft-managed policy:
"Multifactor authentication for admins accessing Microsoft Admin Portals"
but it is in Report-only mode.

Question:
Is Microsoft Entra ID enforcing MFA automatically for privileged roles (like Global Administrator) in admin portals, even when no Conditional Access or PIM policy requires it?

And if so, is there any supported way to fully exclude a break-glass account from this behavior?

Thanks in advance.

Azure Artifact Signing: SignTool "Access is denied" with active Public Trust profile

samuelRiosLazo — Sat, 25 Apr 2026 23:21:29 GMT

I’m blocked on Azure Artifact Signing for Windows EXE signing.

What is already confirmed:

- Account endpoint: https://wus2.codesigning.azure.net/

- Code signing account: notarios

- Certificate profile: notarios-public-trust (Public Trust, Active)

- Identity validation: Completed

- User object id: 9aa27294-c04d-4aab-a7b2-3a8b10be96f9

- RBAC includes:

- Artifact Signing Identity Verifier

- Artifact Signing Certificate Profile Signer

(also assigned at certificate profile scope)

Signing command (signtool 10.0.26100.0 x64 + dlib):

... sign /v /debug /fd SHA256 /tr http://timestamp.acs.microsoft.com /td SHA256 /dlib "<...>\\Azure.CodeSigning.Dlib.dll" /dmdf "C:\temp\metadata-corr.json" "C:\temp\notarial-app-test.exe"

Error every time:

- SignTool Error: Access is denied.

- Number of files successfully Signed: 0

I also tested Azure CLI auth and explicit AccessToken in metadata; same result.

CorrelationId for troubleshooting:

- notarios-20260425-1859

If anyone from Microsoft can check backend logs for that CorrelationId, I’d appreciate the exact reason and remediation.

Azure RBAC Custom Role Best Practices or Common Build Patterns

nicksal — Mon, 20 Apr 2026 18:40:54 GMT

As a platform admin, I want to grant application admins Contributor access while removing their ability to write or delete most Microsoft.Network resource types, with a few exceptions such as Private Endpoints, Network Interfaces, and Application Gateways.

Based on the effective control plane permissions logic, we designed two custom roles. The first role is a duplicate of the Contributor role, but with Microsoft.Network//Write and Microsoft.Network//Delete added to notActions. The second role adds back specific Microsoft.Network operations using wildcarded resource types, such as Microsoft.Network/networkInterfaces/*.

Application Admin Effective Permissions = Role 1 (Contributor - Microsoft.Network) + Role 2 (for example, Microsoft.Network/networkInterfaces/, Microsoft.Network/networkSecurityGroups/, Microsoft.Network/applicationGateways/write, etc.)

I understand that Microsoft RBAC best practices recommend avoiding wildcard (*) operations. However, my team has found that building roles with individual operations is extremely tedious and time-consuming, especially when trying to understand the impact of each operation.

Does anyone have suggestions for a simpler or more maintainable pattern for implementing this type of custom RBAC design?

Legacy SSRS reports after upgrading Azure DevOps Server 2020 to 2022 or 25H2

fujiwaraH2O — Sat, 18 Apr 2026 04:24:17 GMT

We are currently planning an upgrade from Azure DevOps Server 2020 to Azure DevOps Server 2022 or 25H2, and one of our biggest concerns is reporting.

We understand that Microsoft’s recommended direction is to move to Power BI based on Analytics / OData. However, for on-prem environments with a large number of existing SSRS reports, rebuilding everything from scratch would require significant time and effort.

Since Warehouse and Analysis Services are no longer available in newer versions, we would like to understand how other on-prem teams are handling legacy SSRS reporting during and after the upgrade.

Have you rebuilt your reports in Power BI, moved to another reporting approach, or found a practical way to keep existing SSRS reports available during the transition?

Any real-world experience, lessons learned, or recommended approaches would be greatly appreciated.

Excluding break-glass account from MFA Registration Campaign – impact on existing users?

schiachris — Thu, 16 Apr 2026 14:03:10 GMT

Hi everyone,

I'm currently reviewing the configuration of a break-glass (emergency access) account in Microsoft Entra ID and I have a question regarding MFA registration enforcement.

We currently have an Authentication Methods Registration Campaign enabled for all users for quite some time. We identified that the break-glass account is being required to register MFA due to this configuration.

The account is already excluded from all Conditional Access policies that enforce MFA, so the behavior appears to be specifically coming from the registration campaign (Microsoft Authenticator requirement).

Our goal is to exclude this break-glass account from the MFA registration requirement, following Microsoft best practices.

My question is:

If we edit the existing registration campaign and add an exclusion (user or group), could this have any impact on users who are already registered?

Specifically, could it re-trigger the registration process or affect existing MFA configurations?

We want to avoid any unintended impact, considering this campaign has been in place for a long time.

Has anyone implemented a similar exclusion for break-glass accounts within an active registration campaign? Any insights or confirmation would be really helpful.

Thanks in advance!

Running Commands Across VM Scale Set Instances Without RDP/SSH Using Azure CLI Run Command

vdivizinschi — Wed, 15 Apr 2026 11:46:24 GMT

If you’ve ever managed an Azure Virtual Machine Scale Set (VMSS), you’ve likely run into this situation:

You need to validate something across all nodes, such as:

Checking a configuration value
Retrieving logs
Applying a registry change
Confirming runtime settings
Running a quick diagnostic command

And then you realize:

You’re not dealing with two or three machines you’re dealing with 40… 80… or even hundreds of instances.

The Traditional Approach (and Its Limitations)

Historically, administrators would:

Open RDP connections to Windows nodes
SSH into Linux nodes
Execute commands manually on each instance

While this may work for a small number of machines, in real‑world environments such as:

Azure Batch (user‑managed pools)
Azure Service Fabric (classic clusters)
VMSS‑based application tiers

This approach quickly becomes:

Operationally inefficient
Time‑consuming
Sometimes impossible

Especially when:

RDP or SSH ports are blocked
Network Security Groups restrict inbound connectivity
Administrative credentials are unavailable
Network configuration issues prevent guest access

Azure Run Command

To address this, Azure provides a built‑in capability to execute commands inside virtual machines through the Azure control plane, without requiring direct guest OS connectivity. This feature is called Run Command.

You can review the official documentation here:
Run scripts in a Linux VM in Azure using action Run Commands - Azure Virtual Machines | Microsoft Learn
Run scripts in a Windows VM in Azure using action Run Commands - Azure Virtual Machines | Microsoft Learn

Run Command uses the Azure VM Agent installed on the virtual machine to execute PowerShell or shell scripts directly inside the guest OS.

Because execution happens via the Azure control plane, you can run commands even when:

RDP or SSH ports are blocked
NSGs restrict inbound access
Administrative user configuration is broken

In fact, Run Command is specifically designed to troubleshoot and remediate virtual machines that cannot be accessed through standard remote access methods.

Prerequisites & Restrictions.

Before using Run Command, ensure the following:

VM Agent installed and in Ready state
Outbound connectivity from the VM to Azure public IPs over TCP 443 to return execution results.

If outbound connectivity is blocked, scripts may run successfully but no output will be returned to the caller.

Additional limitations include:

Output limited to the last 4,096 bytes
One script execution at a time per VM
Interactive scripts are not supported
Maximum execution time of 90 minutes

Full list of restrictions and limitations are available here:
https://learn.microsoft.com/en-us/azure/virtual-machines/windows/run-command?tabs=portal%2Cpowershellremove#restrictions

Required Permissions (RBAC)

Executing Run Command requires appropriate Azure RBAC permissions.

Action	Permission
List available Run Commands	Microsoft.Compute/locations/runCommands/read
Execute Run Command	Microsoft.Compute/virtualMachines/runCommand/action

The execution permission is included in:

Virtual Machine Contributor role (or higher)

Users without this permission will be unable to execute remote scripts through Run Command.

Azure CLI: az vm vs az vmss

When using Azure CLI, you’ll encounter two similar‑looking commands that behave very differently.

az vm run-command invoke

Used for standalone VMs
Also used for Flexible VM Scale Sets
Targets VMs by name

az vmss run-command invoke

Used only for Uniform VM Scale Sets
Targets instances by numeric instanceId (0, 1, 2, …)

Example: az vmss run-command invoke --instance-id <id>

Unlike standalone VM execution, VMSS instances must be referenced using the parameter "--instance-id" to identify which scale set instance will run the script.

Important: Uniform vs Flexible VM Scale Sets

This distinction is critical when automating Run Command execution.

Uniform VM Scale Sets

Instances are managed as identical replicas
Each instance has a numeric instanceId
Supported by az vmss run-command invoke

Flexible VM Scale Sets

Each instance is a first‑class Azure VM resource
Instance identifiers are VM names, not numbers
az vmss run-command invoke is not supported
Must use az vm run-command invoke per VM

To determine which orchestration mode your VMSS uses:

az vmss show -g "${RG}" -n "${VMSS}" --query "orchestrationMode" -o tsv

Windows vs Linux Targets

Choose the appropriate command ID based on the guest OS:

Windows VMs → RunPowerShellScript
Linux VMs → RunShellScript

Example Scenario - Retrieve Hostname From All VMSS Instances

The following examples demonstrate how to retrieve the hostname from all VMSS instances using Azure CLI and Bash.

Flexible VMSS, Bash (Azure CLI)

RG="<ResourceGroup>" VMSS="<VMSSName>" SUBSCRIPTION_ID="<SubscriptionID>" az account set --subscription "${SUBSCRIPTION_ID}" VM_NAMES=$(az vmss list-instances \ -g "${RG}" \ -n "${VMSS}" \ --query "[].name" \ -o tsv) for VM in $VM_NAMES; do echo "Running on VM: $VM" az vm run-command invoke \ -g "${RG}" \ -n "$VM" \ --command-id RunShellScript \ --scripts "hostname" \ --query "value[0].message" \ -o tsv done

Uniform VMSS, Bash (Azure CLI)

RG="<ResourceGroup>" VMSS="<VMSSName>" SUBSCRIPTION_ID="<SubscriptionID>" az account set --subscription "${SUBSCRIPTION_ID}" INSTANCE_IDS=$(az vmss list-instances -g "${RG}" -n "${VMSS}" --query "[].instanceId" -o tsv) for ID in $INSTANCE_IDS; do echo "Running on instanceId: $ID" az vmss run-command invoke \ -g "${RG}" \ -n "${VMSS}" \ --instance-id "$ID" \ --command-id RunShellScript \ --scripts "hostname" \ --query "value[0].message" \ -o tsv done

Summary

Azure Run Command provides a scalable method to:

Execute diagnostics
Apply configuration changes
Collect logs
Validate runtime settings

…across VMSS instances without requiring RDP or SSH connectivity.

This significantly simplifies operational workflows in large‑scale compute environments such as:

Azure Batch (user‑managed pools)
Azure Service Fabric classic clusters
VMSS‑based application tiers

Excited to share my latest open-source project: KubeCost Guardian

Hlali_Mohamed_Amine — Fri, 10 Apr 2026 15:16:04 GMT

After seeing how many DevOps teams struggle with Kubernetes cost visibility on Azure, I built a full-stack cost optimization platform from scratch.

𝗪𝗵𝗮𝘁 𝗶𝘁 𝗱𝗼𝗲𝘀:
✅ Real-time AKS cluster monitoring via Azure SDK
✅ Cost breakdown per namespace, node, and pod
✅ AI-powered recommendations generated from actual cluster state
✅ One-click optimization actions
✅ JWT-secured dashboard with full REST API

𝗧𝗲𝗰𝗵 𝗦𝘁𝗮𝗰𝗸:
- React 18 + TypeScript + Vite
- Tailwind CSS + shadcn/ui + Recharts
- Node.js + Express + TypeScript
- Azure SDK (@azure/arm-containerservice)
- JWT Authentication + Azure Service Principal

𝗪𝗵𝗮𝘁 𝗺𝗮𝗸𝗲𝘀 𝗶𝘁 𝗱𝗶𝗳𝗳𝗲𝗿𝗲𝗻𝘁:
Most cost tools show you generic estimates. KubeCost Guardian reads your actual VM size, node count, and cluster configuration to generate recommendations that are specific to your infrastructure not averages.
For example, if your cluster has only 2 nodes with no autoscaler enabled, it immediately flags the HA risk and calculates exactly how much you'd save by switching to Spot instances based on your actual VM size.

This project is fully open-source and built for the DevOps community.

⭐ GitHub: https://github.com/HlaliMedAmine/kubecost-guardian

This project represents hours of hard work, and passion.

I decided to make it open-source so everyone can benefit from it 🤝 ,If you find it useful, I’d really appreciate your support .

Your support motivates me to keep building and sharing more powerful projects 👌.

More exciting ideas are coming soon… stay tuned! 🔥.

Pipeline Intelligence is live and open-source real-time Azure DevOps monitoring powered by AI .

Hlali_Mohamed_Amine — Fri, 10 Apr 2026 15:04:34 GMT

Every DevOps team I've worked with had the same problem: Slow pipelines. Zero visibility. No idea where to start. So I stopped complaining and built the solution.

So I built something about it.

⚡ Pipeline Intelligence is a full-stack Azure DevOps monitoring dashboard that:

✅ Connects to your real Azure DevOps organization via REST API
✅ Detects bottlenecks across all your pipelines automatically
✅ Calculates exactly how much time your team is wasting per month
✅ Uses Gemini AI to generate prioritized fixes with ready-to-paste YAML solutions
✅ JWT-secured, Docker-ready, and fully open-source

Tech Stack:
→ React 18 + Vite + Tailwind CSS
→ Node.js + Express + Azure DevOps API v7
→ Google Gemini 1.5 Flash
→ JWT Authentication + Docker

𝗪𝗵𝗮𝘁 𝗺𝗮𝗸𝗲𝘀 𝗶𝘁 𝗱𝗶𝗳𝗳𝗲𝗿𝗲𝗻𝘁?
Most tools show you generic estimates.
Pipeline Intelligence reads your actual cluster config, node count, and pipeline structure and gives you recommendations specific to your infrastructure.

🎯 This year, I set myself a personal challenge:
Build and open-source a series of production-grade tools exclusively focused on Azure services tools that solve real problems for real DevOps teams.

This project represents weeks of research, architecture decisions, and late-night debugging sessions. I'm sharing it with the community because I believe great tooling should be accessible to everyone not locked behind enterprise paywalls.

If this resonates with you, I have one simple ask:
👉 A like, a comment, or a share takes 3 seconds but it helps this reach the DevOps engineers who need it most.

Your support

is what keeps me building. ❤️

GitHub: https://github.com/HlaliMedAmine/pipeline-intelligence

Building a Production-Ready Azure Lighthouse Deployment Pipeline with EPAC

omidvahedv — Thu, 09 Apr 2026 11:24:00 GMT

Recently I worked on an interesting project for an end-to-end Azure Lighthouse implementation.
What really stood out to me was the combination of Azure Lighthouse, EPAC, DevOps, and workload identity federation.
The deployment model was so compelling that I decided to build and validate the full solution hands-on in my own personal Azure tenants.
The result is a detailed article that documents the entire journey, including pipeline design, implementation steps, and the scripts I prepared along the way.
You can read the full article here

Azure Key Vault Replication: Why Paired Regions Alone Don’t Guarantee Business Continuity

joclemen — Mon, 06 Apr 2026 19:45:04 GMT

As customers modernize toward multi‑region architectures in Azure, one question comes up repeatedly:

“If my region goes down, will Azure Key Vault continue to work without disruption?”

The short answer: it depends on what you mean by “work.”

Azure Key Vault provides strong durability and availability guarantees, but those guarantees are often misunderstood—especially when customers assume paired‑region replication equals full disaster recovery. In reality, Azure Key Vault replication is designed for survivability, not uninterrupted write access or customer‑controlled failover.

This post explains:

How Azure Key Vault replication actually works (per Microsoft Learn)
Why paired‑region failover does not equal business continuity
Two reference architectures that implement true multi‑region Key Vault availability, with Terraform

How Azure Key Vault Replication Works (Per Microsoft Learn)

Azure Key Vault includes multiple layers of Microsoft‑managed redundancy.

In‑Region and Zone Resiliency

Vault contents are replicated within the region.
In regions that support availability zones, Key Vault is zone‑resilient by default.
This protects against localized hardware or zone failures.

Paired‑Region Replication

If a Key Vault is deployed in a region with an Azure‑defined paired region, its contents are asynchronously replicated to that paired region.
This replication is automatic and cannot be configured, observed, or tested by customers.

Microsoft‑Managed Regional Failover

If Microsoft declares a full regional outage, requests are automatically routed to the paired region.
After failover, the vault operates in read‑only mode:

✅ Read secrets, keys, and certificates
✅ Perform cryptographic operations
❌ Create, update, rotate, or delete secrets, keys, or certificates

This is a critical distinction.

Paired‑region replication preserves access — not operational continuity.

Why Paired‑Region Replication Is Not Business Continuity

From a reliability and DR perspective, several limitations matter:

Failover is Microsoft‑initiated, not customer‑controlled
No write operations during regional failover
No secret rotation or certificate renewal
No way to test DR
Accidental deletions replicate
No point‑in‑time recovery without backups

Microsoft Learn explicitly states that critical workloads may require custom multi‑region strategies beyond built‑in replication.

For many customers, this means Azure Key Vault becomes a single‑region dependency in an otherwise multi‑region application design.

The Multi‑Region Key Vault Pattern

The two GitHub repositories below implement a common architectural shift:

Multiple independent Key Vaults deployed in separate regions,

with customer‑controlled replication and failover.

Instead of relying on invisible platform replication, the vaults become first‑class, region‑scoped resources, aligned with application failover.

Solution 1: Private, Locked‑Down Multi‑Region Key Vault Replication

Repository:

👉 https://github.com/jclem2000/KeyVault-MultiRegion-Replication-Private

Architecture Highlights

Independent Key Vault per region
Private Endpoints only
No public network exposure
Terraform‑based deployment
Controlled replication using Event Based synchronization

Private Multi-Region Key Vault Replication

What This Enables

✅ Full read/write access during regional outages
✅ Continued secret rotation and certificate renewal
✅ Customer‑defined failover and RTO
✅ DR testing and validation
✅ Strong alignment with zero‑trust and regulated environments

Trade‑offs

Higher operational complexity
Requires automation and application awareness of multiple vaults

Solution 2: Low‑Cost Public Multi‑Region Key Vault Replication

Repository:

👉 https://github.com/jclem2000/KeyVault-MultiRegion-Replication-Public

Architecture Highlights

Independent Key Vault per region
Public endpoints
Minimal networking dependencies
Terraform‑based
Controlled replication using Event Based synchronization
Optimized for simplicity and cost

Low-Cost Multi-Region Key Vault Replication

What This Enables

✅ Full read/write availability in any region
✅ Clear and testable DR posture
✅ Lower cost than private endpoint designs
✅ Suitable for many non‑regulated workloads

Trade‑offs

Public exposure (mitigated via firewall rules, RBAC, and conditional access)
Not appropriate for all compliance requirements
Requires automation and application awareness of multiple vaults

Azure Native Replication vs Customer‑Managed Multi‑Region Vaults

Capability	Azure Paired Region	Multi‑Region Vaults
Read access during outage	✅	✅
Write access during outage	❌	✅
Secret rotation during outage	❌	✅
Customer‑controlled failover	❌	✅
DR testing	❌	✅
Isolation from accidental deletion	❌	✅
Predictable RTO	❌	✅

Azure Key Vault’s native replication optimizes for platform durability.

The multi‑region pattern optimizes for application continuity.

When to Use Each Approach

Paired‑Region Replication Is Often Enough When:

Secrets are mostly static
Read‑only access during outages is acceptable
RTO is flexible
You prefer Microsoft‑managed recovery

Multi‑Region Vaults Are Recommended When:

Secrets or certificates rotate frequently
Applications must remain writable during outages
Deterministic failover is required
DR testing is mandatory
Regulatory or operational isolation is needed

Closing Thoughts

Azure Key Vault behaves exactly as documented on Microsoft Learn—but it’s important to be clear about what those guarantees mean. Paired‑region replication protects your data, not your ability to operate.

If your application is designed to survive a regional outage, Key Vault must follow the same multi‑region design principles as the application itself.

The reference architectures above show how to extend Azure’s native durability model into true operational resilience, without waiting for a platform‑level failover decision.

How on god's green earth do you buy an API key? Bing custom search API.

m3ntosandcoke — Sat, 04 Apr 2026 14:04:18 GMT

I just want to give Microsoft money. I want to buy an API key for the Bing custom search. How do I do this? I get to Production and click "Click to issue paid tier key" and I keep getting the same god-awful

"""

Could not create the marketplace item

Oops!

Could not create the marketplace item

Gallery item is required, no gallery item is provided.

"""

in the Azure marketplace. I just want to spend money. How do I do that?

The March 2026 Innovation Challenge Winners

macalde — Mon, 06 Apr 2026 13:17:23 GMT

For this round of the Innovation Challenge the organizations we sponsor helped over 15,000 developers get the skills it takes to build AI solutions on Azure. This program is grounded in Microsoft’s mission and designed to enable a diverse and qualified community of professional developers coming together to tackle big problems. We helped almost 1,000 people earn Microsoft certifications and Applied Skills credentials, and 300 participated in the invitation only March 2026 Innovation Challenge hackathon. Teams represented SHPE, Women in Cloud, Código Facilito, DIO, GenSpark, NASA Space Apps, Project Blue Mountain, and TechBridge.

Check out the winning project to meet some of the best AI talent in our community and to get inspired about what we can build together!

First place $10,000

Pebble. - AI Cognitive Load Companion

Pebble. is named after a worry stone: something small and smooth you reach for when the world feels like too much. It's an AI cognitive support companion that turns overwhelming documents, tasks, and information into calm, structured clarity. Built for neurodivergent minds. Useful for everyone.

Second place $5,000

The Living Memory Bridge

We believe dementia represents the most extreme form of cognitive overload that exists. It is not just information overload. It is cognitive loss: the gradual erosion of the very tools people use to process the world. Every principle in the brief applies here in its most urgent form: simplified language, adaptive communication, calm and dignity-preserving interactions, personalized memory anchors, and support that meets people exactly where they are.

Query to Insight Analytics CRAM

CRAM is a natural language healthcare analytics platform built entirely on Azure that lets clinical and administrative staff query a patient database using plain English, no SQL required. Users type a question like "What are the top 10 conditions among diabetic patients?" and get back a written summary, a data table, and an auto-generated chart in seconds.

Third place $2,500

ClearStep

ClearStep is an action-first AI system designed to reduce decision overload in high-risk or confusing situations. Instead of only detecting risk, it tells users exactly what to do next. The core innovation is architectural: model output is not trusted. Every response is enforced by a validation layer that guarantees structure, corrects model errors, and prevents unsafe or misleading outputs from reaching the user.

DataTalk

Our platform enables seamless data ingestion from Excel, CSV, SharePoint, and OneDrive, processes it through a two-layer analytical pipeline powered by DuckDB, and orchestrates four specialized AI agents that work as a team: understanding intent, reading data structure, generating and self-correcting SQL, and enforcing security and auditability at every step

RAGulator AI Governance Engine

Advanced, governed, and traceable RAG (Retrieval-Augmented Generation) system for international trade. RAGulator is a 100% functional solution that unifies the Azure intelligence ecosystem to deliver grounded responses with immutable bibliographic citations.

Building Multi-Agent Orchestration Using Microsoft Semantic Kernel: A Complete Step-by-Step Guide

ani_ms_emea — Wed, 01 Apr 2026 11:04:00 GMT

What You Will Build

By the end of this guide, you will have a working multi-agent system where 4 specialist AI agents collaborate to diagnose production issues:

ClientAnalyst — Analyzes browser, JavaScript, CORS, uploads, and UI symptoms
NetworkAnalyst — Analyzes DNS, TCP/IP, TLS, load balancers, and firewalls
ServerAnalyst — Analyzes backend logs, database, deployments, and resource limits
Coordinator — Synthesizes all findings into a root cause report with a prioritized action plan

These agents don't just run in sequence — they debate, cross-examine, and challenge each other's findings through a shared conversation, producing a diagnosis that's better than any single agent could achieve alone.

Why Multi-Agent? The Problem with Single Agents
Architecture Overview
Understanding the Key SK Components
The Actor Model — How InProcessRuntime Works
Setting Up Your Development Environment
Step-by-Step: Building the Multi-Agent Analyzer
The Agent Interaction Flow — Round by Round
Bugs I Found & Fixed — Lessons Learned
Running with Different AI Providers
What to Build Next

1. Why Multi-Agent? The Problem with Single Agents

A single AI agent analyzing a production issue is like having one doctor diagnose everything — they'll catch issues in their specialty but miss cross-domain connections.

Consider this problem: "Users report 504 Gateway Timeout errors when uploading files larger than 10MB. Started after Friday's deployment. Worse during peak hours."

A single agent might say "it's a server timeout" and stop. But the real root cause often spans multiple layers:

The client is sending chunked uploads with an incorrect Content-Length header (client-side bug)
The load balancer has a 30-second timeout that's too short for large uploads (network config)
The server recently deployed a new request body parser that's 3x slower (server-side regression)
The combination only fails during peak hours because connection pool saturation amplifies the latency

No single perspective catches this. You need specialists who analyze independently, then debate to find the cross-layer causal chain. That's what multi-agent orchestration gives you.

The 5 Orchestration Patterns in SK

Semantic Kernel provides 5 built-in patterns for agent collaboration:

SEQUENTIAL:

A → B → C → Done
(pipeline — each builds on previous)

CONCURRENT:

↗ A ↘
Task → B → Aggregate
↘ C ↗
(parallel — results merged)

GROUP CHAT:

A ↔ B ↔ C ↔ D ← We use this one
(rounds, shared history, debate)

HANDOFF:

A → (stuck?) → B → (complex?) → Human
(escalation with human-in-the-loop)

MAGENTIC:

LLM picks who speaks next dynamically
(AI-driven speaker selection)

We use GroupChatOrchestration with RoundRobinGroupChatManager because our problem requires agents to see each other's work, challenge assumptions, and build on each other's analysis across two rounds.

2. Architecture Overview

Here's the complete architecture of what we're building:

3. Understanding the Key SK Components

Before we write code, let's understand the 5 components we'll use and the design pattern each implements:

ChatCompletionAgent — Strategy Pattern

The agent definition. Each agent is a combination of:

name — unique identifier (used in round-robin ordering)
instructions — the persona and rules (this is the prompt engineering)
service — which AI provider to call (Strategy Pattern — swap providers without changing agent logic)
description — what other agents/tools understand about this agent

agent = ChatCompletionAgent( name="ClientAnalyst", instructions="You are ONLY ClientAnalyst...", service=gemini_service, # ← Strategy: swap to OpenAI with zero changes description="Analyzes client-side issues", )

GroupChatOrchestration — Mediator Pattern

The orchestration defines HOW agents interact. It's the Mediator — agents don't talk to each other directly. Instead, the orchestration manages a shared ChatHistory and routes messages through the Manager.

RoundRobinGroupChatManager — Strategy Pattern

The Manager decides WHO speaks next. RoundRobinGroupChatManager cycles through agents in a fixed order. SK also provides AutomaticGroupChatManager where the LLM decides who speaks next.

max_rounds is the total number of messages per agent or cycle. With 4 agents and max_rounds=8, each agent speaks exactly twice.

InProcessRuntime — Actor Model Abstraction

The execution engine. Every agent becomes an "actor" with its own kind of mailbox (message queue). The runtime delivers messages between actors.

Key properties:

No shared state — agents communicate only through messages
Sequential processing — each agent processes one message at a time
Location transparency — same code works in-process today, distributed tomorrow

agent_response_callback — Observer Pattern

A function that fires after EVERY agent response. We use it to display each agent's output in real-time with emoji labels and round numbers.

4. The Actor Model — How InProcessRuntime Works

The Actor Model is a concurrency pattern where each entity is an isolated "actor" with a private mailbox. Here's what happens inside InProcessRuntime when we run our demo:

runtime.start()

│

├── Creates internal message loop (asyncio event loop)

│

orchestration.invoke(task="504 timeout...", runtime=runtime)

│

├── Creates Actor[Orchestrator] → manages overall flow

├── Creates Actor[Manager] → RoundRobinGroupChatManager

├── Creates Actor[ClientAnalyst] → mailbox created, waiting

├── Creates Actor[NetworkAnalyst] → mailbox created, waiting

├── Creates Actor[ServerAnalyst] → mailbox created, waiting

└── Creates Actor[Coordinator] → mailbox created, waiting

Manager receives "start" message

│

├── Checks turn order: [Client, Network, Server, Coordinator]

├── Sends task to ClientAnalyst mailbox

│ → ClientAnalyst processes: calls LLM → response

│ → Response added to shared ChatHistory

│ → callback fires (displayed in Notebook UI)

│ → Sends "done" back to Manager

│

├── Manager updates: turn_index=1

├── Sends to NetworkAnalyst mailbox

│ → Same flow...

│

├── ... (ServerAnalyst, Coordinator for Round 1)

│

├── Manager checks: messages=4, max_rounds=8 → continue

│

├── Round 2: same cycle with cross-examination

│

└── After message 8: Manager sends "complete"

→ OrchestrationResult resolves

→ result.get() returns final answer

runtime.stop_when_idle()

→ All mailboxes empty → clean shutdown

The Actor Model guarantees:

No race conditions (each actor processes one message at a time)
No deadlocks (no shared locks to contend for)
No shared mutable state (agents communicate only via messages)

5. Setting Up Your Development Environment

Prerequisites

Python 3.11 or 3.12 (3.13+ may have compatibility issues with some SK connectors)
Visual Studio Code with the Python and Jupyter extensions
An API key from one of: Google AI Studio (free), OpenAI

Step 1: Install Python

Download from python.org. During installation, check "Add Python to PATH".

Verify:

python --version # Python 3.12.x

Step 2: Install VS Code Extensions

Open VS Code, go to Extensions (Ctrl+Shift+X), and install:

Python (by Microsoft) — Python language support
Jupyter (by Microsoft) — Notebook support
Pylance (by Microsoft) — IntelliSense and type checking

Step 3: Create Project Folder

mkdir sk-multiagent-demo cd sk-multiagent-demo

Open in VS Code:

code .

Step 4: Create Virtual Environment

Open the VS Code terminal (Ctrl+`) and run:

# Create virtual environment python -m venv sk-env # Activate it # Windows: sk-env\Scripts\activate # macOS/Linux: source sk-env/bin/activate

You should see (sk-env) in your terminal prompt.

Step 5: Install Semantic Kernel

For Google Gemini (free tier — recommended for getting started):

pip install semantic-kernel[google] python-dotenv ipykernel

For OpenAI (paid API key):

pip install semantic-kernel openai python-dotenv ipykernel

For Azure AI Foundry (enterprise, Entra ID auth):

pip install semantic-kernel azure-identity python-dotenv ipykernel

Step 6: Register the Jupyter Kernel

python -m ipykernel install --user --name=sk-env --display-name="Semantic Kernel (Python 3.12)"

You can also select if this is already available from your environment from VSCode as below:

Step 7: Get Your API Key

Option A — Google Gemini (FREE, recommended for demo):

Go to https://aistudio.google.com/apikey
Click "Create API Key"
Copy the key

Free tier limits: 15 requests/minute, 1 million tokens/minute — more than enough for this demo.

Option B — OpenAI:

Go to https://platform.openai.com/api-keys
Create a new key
Copy the key

Option C — Azure AI Foundry:

Deploy a model in Azure AI Foundry portal
Note the endpoint URL and deployment name
If key-based auth is disabled, you'll need Entra ID with permissions

Step 8: Create the .env File

In your project root, create a file named .env:

For Gemini:

GOOGLE_AI_API_KEY=AIzaSy...your-key-here GOOGLE_AI_GEMINI_MODEL_ID=gemini-2.5-flash

For OpenAI:

OPENAI_API_KEY=sk-...your-key-here OPENAI_CHAT_MODEL_ID=gpt-4o

For Azure AI Foundry:

AZURE_OPENAI_ENDPOINT=https://your-resource.cognitiveservices.azure.com AZURE_OPENAI_CHAT_DEPLOYMENT_NAME=gpt-4o AZURE_OPENAI_API_KEY=your-key

Step 9: Create the Notebook

In VS Code:

Click File > New File
Save as multi_agent_analyzer.ipynb
In the top-right of the notebook, click Select Kernel
Choose Semantic Kernel (Python 3.12) (or your sk-env)

Your environment is ready. Let's build.

6. Step-by-Step: Building the Multi-Agent Analyzer

Cell 1: Verify Setup

import semantic_kernel print(f"Semantic Kernel version: {semantic_kernel.__version__}") from semantic_kernel.agents import ( ChatCompletionAgent, GroupChatOrchestration, RoundRobinGroupChatManager, ) from semantic_kernel.agents.runtime import InProcessRuntime from semantic_kernel.contents import ChatMessageContent print("All imports successful")

Cell 2: Load API Key and Create Service

For Gemini:

import os from dotenv import load_dotenv load_dotenv() from semantic_kernel.connectors.ai.google.google_ai import ( GoogleAIChatCompletion, GoogleAIChatPromptExecutionSettings, ) from semantic_kernel.contents import ChatHistory GEMINI_API_KEY = os.getenv("GOOGLE_AI_API_KEY") GEMINI_MODEL = os.getenv("GOOGLE_AI_GEMINI_MODEL_ID", "gemini-2.5-flash") service = GoogleAIChatCompletion( gemini_model_id=GEMINI_MODEL, api_key=GEMINI_API_KEY, ) print(f"Service created: Gemini {GEMINI_MODEL}") # Smoke test settings = GoogleAIChatPromptExecutionSettings() test_history = ChatHistory(system_message="You are a helpful assistant.") test_history.add_user_message("Say 'Connected!' and nothing else.") response = await service.get_chat_message_content( chat_history=test_history, settings=settings ) print(f"Model says: {response.content}")

For OpenAI:

import os from dotenv import load_dotenv load_dotenv() from semantic_kernel.connectors.ai.open_ai import ( OpenAIChatCompletion, OpenAIChatPromptExecutionSettings, ) from semantic_kernel.contents import ChatHistory service = OpenAIChatCompletion( ai_model_id=os.getenv("OPENAI_CHAT_MODEL_ID", "gpt-4o"), ) print(f"Service created: OpenAI {os.getenv('OPENAI_CHAT_MODEL_ID', 'gpt-4o')}") # Smoke test settings = OpenAIChatPromptExecutionSettings() test_history = ChatHistory(system_message="You are a helpful assistant.") test_history.add_user_message("Say 'Connected!' and nothing else.") response = await service.get_chat_message_content( chat_history=test_history, settings=settings ) print(f"Model says: {response.content}")

Cell 3: Define All 4 Agents

This is the most important cell — the prompt engineering that makes the demo work:

from semantic_kernel.agents import ChatCompletionAgent # ═══════════════════════════════════════════════════ # AGENT 1: Client-Side Analyst # ═══════════════════════════════════════════════════ client_agent = ChatCompletionAgent( name="ClientAnalyst", description="Analyzes problems from the client-side: browser, JS, CORS, caching, UI symptoms", instructions="""You are ONLY **ClientAnalyst**. You must NEVER speak as NetworkAnalyst, ServerAnalyst, or Coordinator. Every word you write is from ClientAnalyst's perspective only. You are a senior front-end and client-side diagnostics expert. When given a problem statement, analyze it EXCLUSIVELY from the client side: 1. **Browser & Rendering**: DOM issues, JavaScript errors, CSS rendering, browser compatibility, memory leaks, console errors. 2. **Client-Side Caching**: Stale cache, service worker issues, local storage corruption. 3. **Network from Client View**: CORS errors, preflight failures, request timeouts, client-side retry storms, fetch/XHR configuration. 4. **Upload Handling**: File API usage, chunk upload implementation, progress tracking, FormData construction, content-type headers. 5. **UI/UX Symptoms**: What the user sees, error messages displayed, loading states. ROUND 1: Provide your independent analysis. Do NOT reference other agents. List your top 3 most likely causes with evidence. Every response MUST be at least 200 words. ROUND 2: You MUST: - Reference NetworkAnalyst and ServerAnalyst BY NAME - State specifically where you AGREE or DISAGREE with their findings - Answer the Coordinator's questions from your perspective - Add NEW cross-layer insights you see from the client perspective - Do NOT just say 'I agree' — provide substantive technical reasoning Be specific, evidence-based, and prioritize findings by likelihood.""", service=service, ) # ═══════════════════════════════════════════════════ # AGENT 2: Network Analyst # ═══════════════════════════════════════════════════ network_agent = ChatCompletionAgent( name="NetworkAnalyst", description="Analyzes problems from the network side: DNS, TCP, TLS, firewalls, load balancers, latency", instructions="""You are ONLY **NetworkAnalyst**. You must NEVER speak as ClientAnalyst, ServerAnalyst, or Coordinator. Every word you write is from NetworkAnalyst's perspective only. You are a senior network infrastructure diagnostics expert. When given a problem statement, analyze it EXCLUSIVELY from the network layer: 1. **DNS & Resolution**: DNS TTL, propagation delays, record misconfigurations. 2. **TCP/IP & Connections**: Connection pooling, keep-alive, TCP window scaling, connection resets, SYN floods. 3. **TLS/SSL**: Certificate issues, handshake failures, protocol version mismatches. 4. **Load Balancers & Proxies**: Sticky sessions, health checks, timeout configs, request body size limits, proxy buffering. 5. **Firewall & WAF**: Rule blocks, rate limiting, request inspection delays, geo-blocking, DDoS protection interference. ROUND 1: Provide your independent analysis. Do NOT reference other agents. List your top 3 most likely causes with evidence. Every response MUST be at least 200 words. ROUND 2: You MUST: - Reference ClientAnalyst and ServerAnalyst BY NAME - State specifically where you AGREE or DISAGREE with their findings - Answer the Coordinator's questions from your perspective - Add NEW cross-layer insights you see from the network perspective - Do NOT just say 'I am ready to proceed' — provide substantive technical analysis Be specific, evidence-based, and prioritize findings by likelihood.""", service=service, ) # ═══════════════════════════════════════════════════ # AGENT 3: Server-Side Analyst # ═══════════════════════════════════════════════════ server_agent = ChatCompletionAgent( name="ServerAnalyst", description="Analyzes problems from the server side: backend app, database, logs, resources, deployments", instructions="""You are ONLY **ServerAnalyst**. You must NEVER speak as ClientAnalyst, NetworkAnalyst, or Coordinator. Every word you write is from ServerAnalyst's perspective only. You are a senior backend and infrastructure diagnostics expert. When given a problem statement, analyze it EXCLUSIVELY from the server side: 1. **Application Server**: Error logs, exception traces, thread pool exhaustion, memory leaks, CPU spikes, garbage collection pauses. 2. **Database**: Slow queries, connection pool saturation, lock contention, deadlocks, replication lag, query plan changes. 3. **Deployment & Config**: Recent deployments, configuration changes, feature flags, environment variable mismatches, rollback candidates. 4. **Resource Limits**: File upload size limits, request body limits, disk space, temporary file cleanup, storage quotas. 5. **External Dependencies**: Upstream API timeouts, third-party service degradation, queue backlogs, cache (Redis/Memcached) issues. ROUND 1: Provide your independent analysis. Do NOT reference other agents. List your top 3 most likely causes with evidence. Every response MUST be at least 200 words. ROUND 2: You MUST: - Reference ClientAnalyst and NetworkAnalyst BY NAME - State specifically where you AGREE or DISAGREE with their findings - Answer the Coordinator's questions from your perspective - Add NEW cross-layer insights you see from the server perspective - Do NOT just say 'I agree' — provide substantive technical reasoning Be specific, evidence-based, and prioritize findings by likelihood.""", service=service, ) # ═══════════════════════════════════════════════════ # AGENT 4: Coordinator # ═══════════════════════════════════════════════════ coordinator_agent = ChatCompletionAgent( name="Coordinator", description="Synthesizes all specialist analyses into a final root cause report with prioritized action plan", instructions="""You are ONLY **Coordinator**. You must NEVER speak as ClientAnalyst, NetworkAnalyst, or ServerAnalyst. You synthesize — you do NOT do domain-specific analysis. You are the lead engineer who synthesizes the team's findings. ═══ ROUND 1 BEHAVIOR (your first turn, message 4) ═══ Keep this SHORT — maximum 300 words. - Note 2-3 KEY PATTERNS across the three analyses - Identify where specialists AGREE (high-confidence) - Identify where they CONTRADICT (needs resolution) - Ask 2-3 SPECIFIC QUESTIONS for Round 2 Round 1 MUST NOT: assign tasks, create action plans, write reports, or tell agents what to take lead on. Observation + questions ONLY. ═══ ROUND 2 BEHAVIOR (your final turn, message 8) ═══ Keep this FOCUSED — maximum 800 words. Produce a structured report: 1. **Root Cause** (1 paragraph): The #1 most likely cause with causal chain across layers. Reference specific findings from each specialist. 2. **Confidence** (short list): - HIGH: Areas where all 3 agreed - MEDIUM: Areas where 2 of 3 agreed - LOW: Disagreements needing investigation 3. **Action Plan** (numbered, max 6 items): For each: - What to do (specific) - Owner (Client/Network/Server team) - Time estimate 4. **Quick Wins vs Long-term** (2 short lists) Do NOT repeat what specialists already said verbatim. Synthesize, don't echo.""", service=service, ) # ═══════════════════════════════════════════════════ # All 4 agents — order = RoundRobin order # ═══════════════════════════════════════════════════ agents = [client_agent, network_agent, server_agent, coordinator_agent] print(f"{len(agents)} agents created:") for i, a in enumerate(agents, 1): print(f" {i}. {a.name}: {a.description[:60]}...") print(f"\nRoundRobin order: {' → '.join(a.name for a in agents)}")

Cell 4: Run the Analysis

from semantic_kernel.agents import GroupChatOrchestration, RoundRobinGroupChatManager from semantic_kernel.agents.runtime import InProcessRuntime from semantic_kernel.contents import ChatMessageContent from IPython.display import display, Markdown # ╔══════════════════════════════════════════════════════════╗ # ║ EDIT YOUR PROBLEM STATEMENT HERE ║ # ╚══════════════════════════════════════════════════════════╝ PROBLEM = """ Users are reporting intermittent 504 Gateway Timeout errors when trying to upload files larger than 10MB through our web application. The issue started after last Friday's deployment and seems worse during peak hours (2-5 PM EST). Some users also report that smaller file uploads work fine but the progress bar freezes at 85% for large files before timing out. """ # ════════════════════════════════════════════════════════════ agent_responses = [] def agent_response_callback(message: ChatMessageContent) -> None: name = message.name or "Unknown" content = message.content or "" agent_responses.append({"agent": name, "content": content}) emoji = { "ClientAnalyst": "🖥️", "NetworkAnalyst": "🌐", "ServerAnalyst": "⚙️", "Coordinator": "🎯" }.get(name, "🔹") round_num = (len(agent_responses) - 1) // len(agents) + 1 display(Markdown( f"---\n### {emoji} {name} (Message {len(agent_responses)}, Round {round_num})\n\n{content}" )) MAX_ROUNDS = 8 # 4 agents × 2 rounds = 8 messages exactly task = f"""## Problem Statement {PROBLEM.strip()} ## Discussion Rules You are in a GROUP DISCUSSION with 4 members. You can see ALL previous messages. There are exactly 2 rounds. ### ROUND 1 (Messages 1-4): Independent Analysis - ClientAnalyst, NetworkAnalyst, ServerAnalyst: Analyze from YOUR domain only. Give your top 3 most likely causes with evidence and reasoning. - Coordinator: Note patterns across the 3 analyses. Ask 2-3 specific questions. Do NOT assign tasks yet. ### ROUND 2 (Messages 5-8): Cross-Examination & Final Report - ClientAnalyst, NetworkAnalyst, ServerAnalyst: You MUST reference the OTHER specialists BY NAME. State where you agree, disagree, or have new insights. Answer the Coordinator's questions. Provide SUBSTANTIVE analysis. - Coordinator: Produce the FINAL structured report: root cause, confidence levels, prioritized action plan with owners and time estimates. IMPORTANT: Each agent speaks as THEMSELVES only. Never impersonate another agent.""" display(Markdown(f"## Problem Statement\n\n{PROBLEM.strip()}")) display(Markdown(f"---\n## Discussion Starting — {len(agents)} agents, {MAX_ROUNDS} rounds\n")) # Build and run orchestration = GroupChatOrchestration( members=agents, manager=RoundRobinGroupChatManager(max_rounds=MAX_ROUNDS), agent_response_callback=agent_response_callback, ) runtime = InProcessRuntime() runtime.start() result = await orchestration.invoke(task=task, runtime=runtime) final_result = await result.get(timeout=300) await runtime.stop_when_idle() display(Markdown(f"---\n## FINAL CONCLUSION\n\n{final_result}"))

Cell 5: Statistics and Validation

print("═" * 55) print(" ANALYSIS STATISTICS") print("═" * 55) emojis = {"ClientAnalyst": "🖥️", "NetworkAnalyst": "🌐", "ServerAnalyst": "⚙️", "Coordinator": "🎯"} agent_counts = {} agent_chars = {} for r in agent_responses: agent_counts[r["agent"]] = agent_counts.get(r["agent"], 0) + 1 agent_chars[r["agent"]] = agent_chars.get(r["agent"], 0) + len(r["content"]) for agent, count in agent_counts.items(): em = emojis.get(agent, "🔹") chars = agent_chars.get(agent, 0) avg = chars // count if count else 0 print(f" {em} {agent}: {count} msg(s), ~{chars:,} chars (avg {avg:,}/msg)") print(f"\n Total messages: {len(agent_responses)}") total_chars = sum(len(r['content']) for r in agent_responses) print(f" Total analysis: ~{total_chars:,} characters") # Validation print(f"\n Validation:") import re identity_issues = [] for r in agent_responses: other_agents = [a.name for a in agents if a.name != r["agent"]] for other in other_agents: pattern = rf'(?i)as {re.escape(other)}[,:]?\s+I\b' if re.search(pattern, r["content"][:300]): identity_issues.append(f"{r['agent']} impersonated {other}") if identity_issues: print(f" Identity confusion: {identity_issues}") else: print(f" No identity confusion detected") thin = [r for r in agent_responses if len(r["content"].strip()) < 100] if thin: for t in thin: print(f" Thin response from {t['agent']}") else: print(f" All responses are substantive")

Cell 6: Save Report

from datetime import datetime timestamp = datetime.now().strftime("%Y%m%d_%H%M%S") filename = f"analysis_report_{timestamp}.md" with open(filename, "w", encoding="utf-8") as f: f.write(f"# Problem Analysis Report\n\n") f.write(f"**Generated:** {datetime.now().strftime('%Y-%m-%d %H:%M:%S')}\n") f.write(f"**Agents:** {', '.join(a.name for a in agents)}\n") f.write(f"**Rounds:** {MAX_ROUNDS}\n\n---\n\n") f.write(f"## Problem Statement\n\n{PROBLEM.strip()}\n\n---\n\n") for i, r in enumerate(agent_responses, 1): em = emojis.get(r['agent'], '🔹') round_num = (i - 1) // len(agents) + 1 f.write(f"### {em} {r['agent']} (Message {i}, Round {round_num})\n\n") f.write(f"{r['content']}\n\n---\n\n") f.write(f"## Final Conclusion\n\n{final_result}\n") print(f"Report saved to: {filename}")

7. The Agent Interaction Flow — Round by Round

Here's what actually happens during the 8-message orchestration:

Round 1: Independent Analysis (Messages 1-4)

Msg	Agent	What They See	What They Do
1	ClientAnalyst	Problem statement only	Analyzes from client perspective: upload chunking, progress bar freezing at 85%, CORS, content-type headers
2	NetworkAnalyst	Problem + ClientAnalyst's analysis	Gives INDEPENDENT analysis despite seeing msg 1: load balancer timeouts, proxy body size limits, TCP window scaling
3	ServerAnalyst	Problem + msgs 1-2	Gives INDEPENDENT analysis: recent deployment regression, request body parser, thread pool exhaustion, disk space
4	Coordinator	Problem + msgs 1-3	Observes patterns: "All three mention timeout configuration. ClientAnalyst and NetworkAnalyst both point to body size. Question: Was the deployment a backend-only change or did it include infra?"

Round 2: Cross-Examination (Messages 5-8)

Msg	Agent	What They Do
5	ClientAnalyst	"I agree with NetworkAnalyst that the load balancer timeout is likely a factor — the 85% freeze point matches the 30-second LB timeout for a 10MB upload on our average upload speed. However, I disagree with ServerAnalyst about thread pool exhaustion because the UI shows a clean 504, not a connection reset."
6	NetworkAnalyst	"ServerAnalyst's point about the recent deployment is critical — if the new request parser is 3x slower, that would push uploads past the LB timeout. I can confirm the LB has a 30s idle timeout. The fix is both: increase LB timeout AND optimize the parser."
7	ServerAnalyst	"Responding to Coordinator's question: The deployment was backend-only — a new multipart parser using streaming instead of buffered reads. ClientAnalyst is correct that the 504 is from the LB, not the app. The app itself returns 200 after 45 seconds, but the LB kills the connection at 30."
8	Coordinator	Produces final structured report with root cause: "The backend deployment introduced a slower multipart parser (45s vs 15s for 10MB). The load balancer's 30s timeout kills the connection at ~85% progress. Fix: immediate — increase LB timeout to 120s. Short-term — optimize parser. Long-term — implement chunked uploads with progress resumption."

Notice: The Round 2 analysis is dramatically better than Round 1. Agents reference each other by name, build on each other's findings, and the Coordinator can synthesize a cross-layer causal chain that no single agent could have produced.

I made a small adjustment to the issue with Azure Web Apps. Please find the details below from testing carried out using Google Gemini:

8. Bugs I Found & Fixed — Lessons Learned

Building this demo taught me several important lessons about multi-agent systems:

Bug 1: Agents Speaking Only Once

Symptom: Only 4 messages instead of 8.

Root cause: The agents list was missing the Coordinator. It was defined in a separate cell and wasn't included in the members list.

Fix: All 4 agents must be in the same list passed to GroupChatOrchestration.

Bug 2: NetworkAnalyst Says "I'm Ready to Proceed"

Symptom: NetworkAnalyst's Round 2 response was just "I'm ready to proceed with the analysis" — no actual content.

Root cause: The Coordinator's Round 1 message was assigning tasks ("NetworkAnalyst, please check the load balancer config"), and the agent was acknowledging the assignment instead of analyzing.

Fix: Added explicit constraint to Coordinator: "Round 1 MUST NOT assign tasks — observation + questions ONLY."

Bug 3: ServerAnalyst Says "As NetworkAnalyst, I..."

Symptom: ServerAnalyst's response started with "As NetworkAnalyst, I believe..."

Root cause: LLM identity bleeding. When agents share ChatHistory, the LLM sometimes loses track of which agent it's currently playing. This is especially common with Gemini.

Fix: Identity anchoring at the very top of every agent's instructions: "You are ONLY ServerAnalyst. You must NEVER speak as ClientAnalyst, NetworkAnalyst, or Coordinator."

Bug 4: Gemini Gives Thin/Empty Responses

Symptom: Some agents responded with just one sentence or "I concur."

Root cause: Gemini 2.5 Flash is more concise than GPT-4o by default. Without explicit length requirements, it takes shortcuts.

Fix: Added "Every response MUST be at least 200 words" and "Answer the Coordinator's questions" to every specialist's instructions.

Bug 5: Coordinator's Report is 18K Characters

Symptom: The Coordinator's Round 2 response was absurdly long — repeating everything every specialist said.

Fix: Added word limits: "Round 1 max 300 words, Round 2 max 800 words" and "Synthesize, don't echo."

Bug 6: MAX_ROUNDS Math

Symptom: With MAX_ROUNDS=9, ClientAnalyst spoke a 3rd time after the Coordinator's final report — breaking the clean 2-round structure.

Fix: MAX_ROUNDS must equal (number of agents × number of rounds). For 4 agents × 2 rounds = 8.

9. Running with Different AI Providers

The beauty of SK's Strategy Pattern is that you change ONE LINE to switch providers. Everything else — agents, orchestration, callbacks, validation — stays identical.

Gemini setup:

from semantic_kernel.connectors.ai.google.google_ai import GoogleAIChatCompletion service = GoogleAIChatCompletion( gemini_model_id="gemini-2.5-flash", api_key=os.getenv("GOOGLE_AI_API_KEY"), )

OpenAI Setup

from semantic_kernel.connectors.ai.open_ai import OpenAIChatCompletion service = OpenAIChatCompletion( ai_model_id="gpt-4o", api_key=os.getenv("OPEN_AI_API_KEY"), )

10. What to Build Next

Add Plugins to Agents

Give agents real tools — not just LLM reasoning - looks exciting right ;)

class NetworkDiagnosticPlugin: (description="Pings a host and returns latency") def ping(self, host: str) -> str: result = subprocess.run(["ping", "-c", "3", host], capture_output=True, text=True) return result.stdout class LogSearchPlugin: (description="Searches server logs for error patterns") def search_logs(self, pattern: str, hours: int = 1) -> str: # Query your log aggregator (Splunk, ELK, Azure Monitor) return query_logs(pattern, hours)

Add Filters for Governance

Intercept every agent call for PII redaction and audit logging:

.filter(filter_type=FilterTypes.FUNCTION_INVOCATION) async def audit_filter(context, next): print(f"[AUDIT] {context.function.name} called by agent") await next(context) print(f"[AUDIT] {context.function.name} returned")

Try Different Orchestration Patterns

Replace GroupChat with Sequential for a pipeline approach:

# Instead of debate, each agent builds on the previous orchestration = SequentialOrchestration( members=[client_agent, network_agent, server_agent, coordinator_agent] )

Or Concurrent for parallel analysis:

# All specialists analyze simultaneously, Coordinator aggregates orchestration = ConcurrentOrchestration( members=[client_agent, network_agent, server_agent] )

Deploy to Azure

Move from InProcessRuntime to Azure Container Apps for production scaling. The agent code doesn't change — only the runtime.

Summary

The key insight from building this demo: multi-agent systems produce better results than single agents not because each agent is smarter, but because the debate structure forces cross-domain thinking that a single prompt can never achieve. The Coordinator's final report consistently identifies causal chains that span client, network, and server layers — exactly the kind of insight that production incident response teams need.

Semantic Kernel makes this possible with clean separation of concerns: agents define WHAT to analyze, orchestration defines HOW they interact, the manager defines WHO speaks when, the runtime handles WHERE it executes, and callbacks let you OBSERVE everything. Each piece is independently swappable — that's the power of SK from Microsoft.

Resources:

GitHub: github.com/microsoft/semantic-kernel
Docs: learn.microsoft.com/semantic-kernel
Orchestration Patterns: learn.microsoft.com/semantic-kernel/frameworks/agent/agent-orchestration
Discord: aka.ms/sk/discord

Disclaimer:

The sample scripts provided in this article are provided AS IS without warranty of any kind. The author is not responsible for any issues, damages, or problems that may arise from using these scripts. Users should thoroughly test any implementation in their environment before deploying to production. Azure services and APIs may change over time, which could affect the functionality of the provided scripts. Always refer to the latest Azure documentation for the most up-to-date information.

Thanks for reading this blog! I hope you found it helpful and informative for building AI agents with SK (Semantic Kernel) 😀

Recovering our Default Azure Directory

GMDAL — Tue, 31 Mar 2026 15:55:10 GMT

Hello, everyone, relative newcomer to Azure here. I'm dealing with an inherited situation and, to add to the fun, I've just discovered my organization only has a Basic support plan, so no access to Azure technical support. I'm hoping some knowledgeable souls on here are in a charitable mood and will point me in the right direction.

We're having problems getting to our DNS subscription because it's locked away behind an Azure directory to which we don't seem to have access, and I'm not quite sure this is completely an Azure problem. I was able to get into this directory around a year ago but I so seldomly access it that I'm not sure when this changed.

We have two Azure directories. One is our "regular" directory, named for our organization, and it's linked (not sure of the terminology here) to our domain. Let's call it This.Domain.com. There are no subscriptions in this directory.

The other is named "Default Directory" and it's linked to an onmicrosoft domain -- let's call it OldAdminThisDomain.onmicrosoft.com. When I try to switch to this directory I'm prompted to log in, then I'm hit with the MFA prompt. This is normally not a problem but it's like the MFA was set up for a different account with the same email address. By contrast, I can log into both the regular Azure directory and the 365 admin page with no problem -- I type in my email address (let's call it email address removed for privacy reasons), MFA comes up, and I have several authentication methods to choose from: Microsoft's MFA app, SMS, email, YubiKey, phone, etc., and all these options work.

When trying to log into the Azure Default Directory, however, the MFA acknowledges only either the Microsoft Authenticator app or Use a Verification Code (which also goes through the Microsoft Authenticator app), and neither option yields any prompt on my phone. I seem to recall I effectively had two different "accounts" that somehow used the same email address but had different MFA setups, but again this was around a year and 3 phones ago so I don't have a solid memory of what was happening. I am also aware that, while this should not be permittable, there have been several cases where multiple Microsoft accounts were somehow created using the same email address.

So this is where I am. Ideally we could merge the two Azure directories so that we combine the accessibility of the "regular" directory with the subscription(s?) that are in the Default Directory. Barring that I would have to somehow get the (suspected) two Microsoft accounts based on the email address removed for privacy reasons email address corrected. Any help would be greatly appreciated.

Thanks to all in advance

Docker Engine v29 on Linux: Why data-root No Longer Prevents OS Disk Growth (and How to Fix It)

vdivizinschi — Mon, 23 Mar 2026 18:17:37 GMT

Scope

Applies to Linux hosts only

Does not apply to Windows or Docker Desktop

Problem Summary

After upgrading to Docker Engine v29 or reimaging Linux nodes with this version, you may observe unexpected growth on the OS disk, even when Docker is configured with a custom data-root pointing to a mounted data disk.

This commonly affects cloud environments (VMSS, Azure Batch, self‑managed Linux VMs) where the OS disk is intentionally kept small and container data is expected to reside on a separate data disk.

What Changed in Docker Engine v29 (Linux)

Starting with Docker Engine 29.0, containerd’s image store becomes the default storage backend on fresh installations.

Docker explicitly documents this behavior:

“The containerd image store is the default storage backend for Docker Engine 29.0 and later on fresh installations.”
Docker containerd image store documentation

Key points on Linux:

Docker now delegates image and snapshot storage to containerd
containerd uses its own content store and snapshotters
Docker’s traditional data-root setting no longer controls all container storage

Docker Engine v29 was released on 11 November 2025, and this behavior is by design, not a regression.

Where Disk Usage Goes on Linux

Docker’s daemon documentation clarifies the split:

Legacy storage (pre‑v29 or upgraded installs):
- All data under /var/lib/docker
Docker Engine v29 (containerd image store enabled):
- Images & snapshots → /var/lib/containerd
- Other Docker data (volumes, configs, metadata) → /var/lib/docker

Crucially:

“The data-root option does not affect image and container data stored in /var/lib/containerd when using the containerd image store.”
Docker daemon data directory documentation

This explains why OS disk usage continues to grow even when data-root is set to a data disk.

Why the Old Configuration Worked Before

On earlier Docker versions, Docker fully managed image and snapshot storage. Configuring:

{

"data-root": "/mnt/docker-data"

}

Was sufficient to redirect all container storage off the OS disk.

With Docker Engine v29:

containerd owns image and snapshot storage
data-root only affects Docker‑managed data
OS disk growth after upgrades or reimages is expected behavior

This aligns fully with Docker’s documented design changes.

Linux Workaround: Redirect containerd Storage

To restore the intended behavior on Linux, keeping both Docker and containerd storage on the mounted data disk, containerd’s storage path must also be redirected.

A practical workaround is to relocate /var/lib/containerd using a symbolic link.

Example (Linux)

sudo systemctl stop docker.socket docker containerd || true;

sudo mkdir -p /mnt/docker-data /mnt/containerd;

sudo rm -rf /var/lib/containerd;

sudo ln -s /mnt/containerd /var/lib/containerd;

echo "{\"data-root\": \"/mnt/docker-data\"}" | sudo tee /etc/docker/daemon.json;

sudo systemctl daemon-reload;

sudo systemctl start containerd docker'

What This Does

Stops Docker and containerd
Creates container storage directories on the mounted data disk
Redirects /var/lib/containerd → /mnt/containerd
Keeps Docker’s data-root at /mnt/docker-data
Restarts services with a unified storage layout

This workaround is effective because it explicitly accounts for containerd‑managed paths introduced in Docker Engine v29, restoring the behavior that existed prior to the change.

Key Takeaways

Docker Engine v29 introduces a fundamental storage architecture change on Linux
data-root alone is no longer sufficient
OS disk growth after upgrades or reimages is expected
containerd storage must also be redirected
The workaround aligns with Docker’s official documentation and design

References

Docker daemon data directory
https://docs.docker.com/engine/daemon/
containerd image store (Docker Engine v29)
https://docs.docker.com/engine/storage/containerd/
Docker Engine v29 release notes
https://docs.docker.com/engine/release-notes/29/

AI-102 Develop computer vision solutions in Azure (deprecated)

AnSoMo28 — Sat, 21 Mar 2026 22:05:03 GMT

I have my AI-102 certification exam next week, but Microsoft Learn shows the following:
Develop computer vision solutions in Azure (deprecated)
Does that mean that section won't be covered on the exam?

Azure topics

Cloud-Native vs. Hybrid for the 2026 Workplace

Azure Automation Hybrid Runbook Worker Supported OS

Patterns for low-code Azure config state snapshot + recovery solution for resource groups

Using Github Copilot from Azure Subscription

MFA required for Global Admin without Conditional Access or PIM enforcement

Azure Artifact Signing: SignTool "Access is denied" with active Public Trust profile

Azure RBAC Custom Role Best Practices or Common Build Patterns

Legacy SSRS reports after upgrading Azure DevOps Server 2020 to 2022 or 25H2

Excluding break-glass account from MFA Registration Campaign – impact on existing users?

Running Commands Across VM Scale Set Instances Without RDP/SSH Using Azure CLI Run Command

The Traditional Approach (and Its Limitations)

Azure Run Command

Prerequisites & Restrictions.

Required Permissions (RBAC)

Azure CLI: az vm vs az vmss

Important: Uniform vs Flexible VM Scale Sets

Uniform VM Scale Sets

Flexible VM Scale Sets

Windows vs Linux Targets

Example Scenario - Retrieve Hostname From All VMSS Instances

Flexible VMSS, Bash (Azure CLI)

Uniform VMSS, Bash (Azure CLI)

Summary

Excited to share my latest open-source project: KubeCost Guardian

Pipeline Intelligence is live and open-source real-time Azure DevOps monitoring powered by AI .

Building a Production-Ready Azure Lighthouse Deployment Pipeline with EPAC

Azure Key Vault Replication: Why Paired Regions Alone Don’t Guarantee Business Continuity

How on god's green earth do you buy an API key? Bing custom search API.

The March 2026 Innovation Challenge Winners

Building Multi-Agent Orchestration Using Microsoft Semantic Kernel: A Complete Step-by-Step Guide

What You Will Build

Table of Contents

1. Why Multi-Agent? The Problem with Single Agents

The 5 Orchestration Patterns in SK

2. Architecture Overview

3. Understanding the Key SK Components

ChatCompletionAgent — Strategy Pattern

GroupChatOrchestration — Mediator Pattern

RoundRobinGroupChatManager — Strategy Pattern

InProcessRuntime — Actor Model Abstraction

agent_response_callback — Observer Pattern

4. The Actor Model — How InProcessRuntime Works

5. Setting Up Your Development Environment

Prerequisites

Step 1: Install Python

Step 2: Install VS Code Extensions

Step 3: Create Project Folder

Step 4: Create Virtual Environment

Step 5: Install Semantic Kernel

Step 6: Register the Jupyter Kernel

Step 7: Get Your API Key

Step 8: Create the .env File

Step 9: Create the Notebook

6. Step-by-Step: Building the Multi-Agent Analyzer

7. The Agent Interaction Flow — Round by Round

Round 1: Independent Analysis (Messages 1-4)

Round 2: Cross-Examination (Messages 5-8)

8. Bugs I Found & Fixed — Lessons Learned

Bug 1: Agents Speaking Only Once

Bug 2: NetworkAnalyst Says "I'm Ready to Proceed"

Bug 3: ServerAnalyst Says "As NetworkAnalyst, I..."

Bug 4: Gemini Gives Thin/Empty Responses

Bug 5: Coordinator's Report is 18K Characters

Bug 6: MAX_ROUNDS Math

9. Running with Different AI Providers

10. What to Build Next

Add Plugins to Agents

Add Filters for Governance

Try Different Orchestration Patterns

Deploy to Azure

Summary

Disclaimer:

Recovering our Default Azure Directory

Docker Engine v29 on Linux: Why data-root No Longer Prevents OS Disk Growth (and How to Fix It)

Scope

Problem Summary

What Changed in Docker Engine v29 (Linux)

Where Disk Usage Goes on Linux

Why the Old Configuration Worked Before