rss.livelink.threads-in-node

Deploying an Agentic Service to Microsoft 365 Copilot with Delegated OBO Access

JamesN — Fri, 24 Apr 2026 13:00:00 GMT

You've built an agentic service. It works. It uses the framework you chose — maybe LangChain, maybe Semantic Kernel, maybe the Microsoft Agent Framework, maybe something entirely your own. It talks to your LLM, calls your downstream APIs, and manages multi-turn conversations the way you designed it.

Now someone asks: "Can we surface this in Microsoft 365 Copilot?"

The obvious path is to rebuild the agent using M365 Copilot Agents (declarative or custom engine). But that means giving up control — over your orchestration logic, your framework choices, your session management, and the way you call downstream services on behalf of the signed-in user.

This guide is for the other path. The one where you keep your existing agentic service largely intact and place an M365 Gateway in front of it — a stateless service that handles the Bot Framework protocol, validates channel-facing tokens, performs the first On-Behalf-Of token exchange, and translates Copilot conversations into your service's native API. Your service can stay Bot- and Teams-agnostic, but if it owns downstream delegated access it should still validate the inbound service token at its own boundary and use that validated assertion for its own OBO chain.

When to use this pattern:

You have (or want to build) your own agentic implementation in your preferred technology and framework
M365 Copilot Agents — declarative or custom engine — don't give you enough control over orchestration, tool calling, or downstream auth
You want total ownership of the agentic logic: pro-code, your LLM, your prompts, your tools, your session store
You need user-delegated access to downstream services (databases, APIs, MCP servers) via chained OBO flows, and you want to own that token chain end-to-end

What follows is a general development guideline — framework-agnostic, language-agnostic — for deploying any agentic service behind Azure Container Apps, exposing it to M365 Copilot through an M365 Gateway, and using chained OBO flows to call downstream services as the signed-in user.

Architecture Overview

The architecture separates concerns into two independently deployable services:

Layer	Responsibility	State
M365 Copilot	User identity, SSO, conversation UX	None (platform)
Gateway	Bot protocol adapter, channel auth, OBO #1 token exchange	Stateless
Agentic Service	Business logic, service-token validation, downstream OBO, session memory	Stateful
Downstream	Data, APIs, MCP servers — called as the delegated user	External

Why two services?

Separation of trust boundaries — The gateway handles Bot Framework protocol and channel auth. The service owns business/data access, validates the service bearer it receives, and never needs Bot credentials.
Independent scaling — The gateway can scale to many replicas. The service (with in-memory sessions) runs as a single replica for MVP.
Framework freedom — The service can use any agentic framework, LLM provider, or custom logic. The gateway is just an HTTP forwarder.
Adapt without modifying — If you already have an agentic service, you can bring it to M365 Copilot by writing only the gateway + service client adapter. The existing service stays untouched.

How reusable is the gateway?

In practice, most of the gateway can be reused across multiple agentic services that need to surface in M365 Copilot.

Reusable across services:

the POST /api/messages host and Bot adapter bootstrap
Bot/channel auth wiring
OBO #1 token acquisition for the downstream service scope
conversation ID to service session ID mapping
long-running turn handling and delayed acknowledgement behavior
busy-turn rejection for overlapping requests in the same conversation
generic seller-safe auth and transient failure handling
ACA, Azure Bot, and app-package deployment shape

Usually service-specific:

the downstream service client
activity-to-service payload translation
the target service scope and app-registration values
service-specific telemetry labels and fallback copy
any session bootstrap contract such as create_session() versus direct first-turn POSTs

The Daily Account Planner MVP follows exactly this split: the wrapper is mostly generic, while the planner client and a small amount of message shaping remain service-specific.

Token Flow Deep Dive

The user's identity flows end-to-end through two chained OBO exchanges. No service ever stores or caches user passwords.

Key principles

The user token never leaves the OBO chain. Each service receives a scoped assertion and exchanges it for the next hop. No service sees the original password.
JWT validation is non-negotiable — and split by boundary. The gateway must validate the Bot/channel-facing token it receives from Microsoft 365. The service must validate the inbound service-scoped bearer before using it for OBO, session ownership, or user scoping.
Internal-only service ingress is recommended hardening, not the only safe option. In ACA, private ingress is a strong default. If the service remains externally reachable, it must still enforce its own bearer validation and authorization checks exactly the same way.
ContextVar carries the validated assertion. The service binds the validated JWT assertion to an async-safe ContextVar in middleware, so any downstream OBO call within the same request automatically picks it up.

Entra ID App Registrations

You need two Entra app registrations plus knowledge of your downstream API's resource app ID.

Step-by-step registration

1. Agentic Service app

Setting	Value
Display name	your-service-api
Identifier URI	api://<service-client-id>
Sign-in audience	AzureADMyOrg (single tenant)
requestedAccessTokenVersion	2
Exposed scope	access_as_user (delegated, User consent)
Required permissions	Downstream API user_impersonation (delegated)
Client secret	Yes — needed for MSAL ConfidentialClient OBO

2. Gateway / Bot app

Setting	Value
Display name	your-bot
Identifier URI	api://botid-<bot-client-id> (Bot SSO convention)
Sign-in audience	AzureADMyOrg
requestedAccessTokenVersion	2
Exposed scope	access_as_user (delegated)
Required permissions	Service app access_as_user (delegated)
Redirect URI	https://token.botframework.com/.auth/web/redirect
Preauthorized clients	Teams Desktop (1fec8e78-...) and Teams Web (5e3ce6c0-...)
Client secret	Yes — needed for Bot Framework auth

3. Admin consent (mandatory)

Both delegated permission grants require tenant admin consent:

Gateway/Bot app → Service app access_as_user ← admin consent
Service app → Downstream user_impersonation ← admin consent

Without admin consent, OBO calls return AADSTS65001: The user or administrator has not consented to use the application.

Operational guidance: if you package this into an operator bootstrap, treat missing admin consent as a blocking failure. Do not continue with service deployment and hope sign-in will work later. Failing fast here produces a much more repeatable operator experience.

4. Azure Bot OAuth connection

Create an OAuth connection on the Azure Bot resource so the Microsoft Agents SDK can perform silent token exchange for the gateway-to-service hop:

az bot authsetting create \ -g $RESOURCE_GROUP \ -n $BOT_RESOURCE_NAME \ -c SERVICE_CONNECTION \ --service Aadv2 \ --client-id $BOT_APP_ID \ --client-secret $BOT_APP_PASSWORD \ --provider-scope-string "$SERVICE_API_SCOPE offline_access openid profile" \ --parameters TenantId="$TENANT_ID" TokenExchangeUrl="api://botid-$BOT_APP_ID"

Component Implementation Guide

Component 1: The M365 Gateway (Stateless, Protocol Adapter)

The gateway's job is to bridge the Bot Framework protocol into whatever HTTP contract your agentic service exposes. It receives Bot activities, validates the channel-facing token path, exchanges the user's SSO token for a service-scoped token (OBO #1), and translates the call into the service's native API shape.

Key insight: If you already have a stateful agentic service — even one that wasn't designed for M365 — the gateway can adapt to it. You do not need to modify your existing service to conform to a specific API contract. The gateway absorbs the M365 protocol translation. The service should still validate the service-scoped bearer that reaches its own API boundary.

Token validation responsibilities in the Gateway

This is a hard requirement, not optional. The gateway is the public-facing trust boundary for the Bot/channel request. If channel-token validation is skipped or incomplete, unauthenticated callers may be able to reach your forwarding path.

The gateway must validate the Bot/channel-facing token according to the SDK/Bot Framework requirements for the channel it serves. After OBO #1 produces a service-scoped bearer, the gateway may inspect that token for diagnostics, routing, or optional metadata forwarding, but the service must remain the authoritative validator for that service token.

Good gateway behavior:

Validate inbound channel traffic using the Bot/Agents SDK middleware or equivalent issuer/signature/audience checks for the incoming channel token
Acquire the service-scoped token for your service API with OBO #1
Forward the service bearer as-is in Authorization: Bearer ...
Optionally extract claims for logging or convenience headers:

– oid (user object ID)

– tid (tenant ID)

– upn or preferred_username

Treat forwarded headers as non-authoritative metadata. They can help with telemetry, correlation, or debugging, but the service should derive identity from the validated bearer it receives.

Adapting to your service's API contract

The gateway contains a service client — a small HTTP adapter class that translates between the Copilot conversation model and your service's native API. This is where you map:

Copilot concept	Your service's equivalent	Mapped in the service client
conversation.id	Session/thread/chat ID	Map on create or first message
User message text	Request body field (could be text, message, prompt, input, etc.)	Reshape the request payload
Reply text	Response field (could be reply, response, output, content, etc.)	Extract from response payload
Authentication	Bearer token, API key, or custom header	Attach the service-scoped delegated token in the right header/field

Example: adapting to different service shapes

# Service client for a service with POST /chat/{thread_id} class MyServiceClient: async def send_turn(self, session_id: str, text: str, token: str) -> str: resp = await self.http.post( f"{self.base_url}/chat/{session_id}", json={"prompt": text, "stream": False}, headers={"Authorization": f"Bearer {token}"}, ) return resp.json()["response"] # extract from service's response shape # Service client for a service with POST /v1/conversations/{id}/messages class AnotherServiceClient: async def send_turn(self, session_id: str, text: str, token: str) -> str: resp = await self.http.post( f"{self.base_url}/v1/conversations/{session_id}/messages", json={"content": text, "role": "user"}, headers={"X-Api-Token": token}, ) return resp.json()["choices"][0]["message"]["content"]

The gateway message handler remains the same regardless of the service client shape:

async def handle_message(context): token = await agent_auth.get_token(context) session_id = context.activity.conversation.id reply = await service_client.send_turn(session_id, context.activity.text, token) await context.send_activity(reply)

What the gateway needs from any service

As long as the service supports these three capabilities — in whatever API shape — the gateway can adapt to it:

Session/thread identity — Some way to maintain conversation state across turns (session ID, thread ID, conversation ID, etc.)
Message exchange — An endpoint that accepts user text and returns a reply
Authentication — Accepts a bearer token or other credential that the gateway can supply via OBO #1

The service does not need to use a specific URL pattern, request/response schema, or framework. The gateway's service client class is the single place where you encode these mappings. The service should still validate the bearer it receives before it trusts any user identity implied by the call.

Key implementation details

Auth handlers — The gateway needs two auth handler paths:

Agentic path (AgenticUserAuthorization): Used when the activity arrives through the M365 Copilot agentic channel. Requires both abs_oauth_connection_name and obo_connection_name.
Connector path (UserAuthorization): Used when the activity arrives through standard Teams / Bot connector. Requires abs_oauth_connection_name only.

auth_handlers = { "service_agentic": AuthHandler( auth_type="AgenticUserAuthorization", abs_oauth_connection_name="SERVICE_CONNECTION", obo_connection_name="SERVICE_OBO_CONNECTION", # may be same as abs scopes=["api://<service-client-id>/access_as_user"], ), "service_connector": AuthHandler( auth_type="UserAuthorization", abs_oauth_connection_name="SERVICE_CONNECTION", obo_connection_name="", scopes=["api://<service-client-id>/access_as_user"], ), }

Invoke handling — Copilot sends invoke activities during the SSO token exchange handshake. The gateway must handle these gracefully (return without error) or the sign-in flow breaks.

Session mapping — Use context.activity.conversation.id as the session key when forwarding to the service. This ensures the same Copilot conversation always maps to the same agentic session.

Healthz bypass — The ACA health probe hits /healthz. Bypass JWT middleware for this path or the probe fails.

Long-running compatibility bridge — If the SDK's built-in long-running proactive path does not preserve the message contract your wrapper needs, keep long-running mode enabled but own a very small gateway-local compatibility bridge. That bridge should preserve the original user message activity while still using the proactive continuation context for the outbound reply. Keep this as an infrastructure concern in the gateway; do not move business logic there.

Technology choices

The gateway uses:

microsoft-agents-hosting-fastapi for the Bot SDK adapter
microsoft-agents-authentication-msal for MSAL-based token exchange
httpx for forwarding HTTP calls to the service

You can substitute any Bot SDK or language as long as you handle the same protocol.

Component 2: The Agentic Service (Stateful, Framework-Agnostic)

The service is a regular HTTP API. It can know nothing about Bot Framework, Teams, or Copilot-specific activity shapes, but if it owns downstream delegated access it should validate the inbound service bearer at its own boundary. It receives the service-scoped token, validates it, binds user claims from that validated token, runs your agentic logic, and returns a reply.

Service trust model: The service is the business/data trust boundary. It should trust the validated bearer token it receives, not convenience headers alone. Forwarded X-User-* headers are optional metadata and should never be the sole basis for user scoping or downstream OBO.

Minimum service requirements

The service needs only these capabilities — everything else is optional:

Requirement	What it means	Why
Validate inbound bearer	Read and validate the Authorization: Bearer header	Needed before using the token for OBO, ownership, or authorization
Extract claims from the validated token	Decode oid, tid, upn / preferred_username, scopes, etc.	Needed for session owner isolation and audit trail
Session/thread identity	Maintain conversation state across turns	Copilot expects multi-turn conversations
Message exchange	Accept user text, return a reply	Core function
OBO #2 (if user-delegated access needed)	MSAL acquire_token_on_behalf_of() with the forwarded assertion	Only if calling downstream APIs as the user
Ingress hardening	Prefer internal-only ingress or equivalent network controls when practical	Reduces attack surface, but does not replace service-side token validation

ContextVar pattern for the OBO assertion

The validated JWT assertion (the raw token string from the gateway) must be available deep in the call stack when a downstream OBO call happens — potentially several layers below the HTTP handler. Use contextvars.ContextVar rather than passing it through every function signature:

from contextvars import ContextVar, Token as CtxToken _USER_ASSERTION: ContextVar[str | None] = ContextVar("user_assertion", default=None) _USER_CLAIMS: ContextVar[TokenClaims | None] = ContextVar("user_claims", default=None) def bind_identity(assertion: str, claims: TokenClaims): return _USER_ASSERTION.set(assertion), _USER_CLAIMS.set(claims) def reset_identity(a_tok: CtxToken, c_tok: CtxToken): _USER_ASSERTION.reset(a_tok) _USER_CLAIMS.reset(c_tok) # In middleware, after validating the bearer: a_tok, c_tok = bind_identity(raw_jwt, validated_claims) try: response = await call_next(request) finally: reset_identity(a_tok, c_tok) # always clean up

Why ContextVar? — It is async-safe (each concurrent request gets its own scope), framework-agnostic (works with FastAPI, Flask, Django, raw asyncio), and avoids threading the assertion through your entire agentic stack.

OBO #2 to downstream services

When your agentic logic needs to call a downstream API as the user:

import msal app = msal.ConfidentialClientApplication( client_id=SERVICE_CLIENT_ID, client_credential=SERVICE_CLIENT_SECRET, authority=f"https://login.microsoftonline.com/{TENANT_ID}", ) result = app.acquire_token_on_behalf_of( user_assertion=_USER_ASSERTION.get(), # from ContextVar scopes=["<downstream-resource-id>/.default"], # e.g. Graph, Databricks, your own API ) downstream_token = result["access_token"]

Common downstream scope examples:

Downstream	Scope
Microsoft Graph	https://graph.microsoft.com/.default
Azure Databricks	2ff814a6-3304-4ab8-85cb-cd0e6f879c1d/.default
Azure SQL	https://database.windows.net/.default
Custom internal API	api://<their-app-id>/.default
MCP server behind Entra	api://<mcp-app-id>/.default

The pattern is identical for any Entra-protected resource — only the scope changes.

Session management

Minimum viable session store for MVP (in-memory, single-replica):

Create — POST /api/chat/sessions → returns session_id
Send message — POST /api/chat/sessions/{id}/messages → returns reply
Get session — GET /api/chat/sessions/{id} → returns turns history
Owner isolation — Every session has an owner_id from the validated JWT oid claim. Reject cross-user access with 403.
Turn windowing — Cap stored turns at max_turns * 2 entries to prevent unbounded memory growth.
Concurrency lock — asyncio.Lock() per session prevents interleaved turns.

Your agentic logic goes here

The service API is a clean boundary. Inside it, use whatever you want:

Microsoft Agent Framework — HandoffBuilder, ConcurrentBuilder, etc.
LangChain / LangGraph — chains, graphs, tool calling
Semantic Kernel — planners and plugins
Custom code — direct OpenAI / Azure OpenAI SDK calls with tool loops
Any other framework — CrewAI, AutoGen, etc.

Reference API contract

If building a new service from scratch, this contract works well with the gateway:

POST /api/chat/sessions/{session_id}/messages Authorization: Bearer <service-scoped-token> Content-Type: application/json {"text": "user message"} → 200 {"session_id": "...", "reply": "...", "turns": [...]} → 401 (bad/missing token) → 404 (session not found — gateway auto-creates and retries)

However, if you have an existing service with a different contract, you do not need to change it. Instead, adapt the gateway's service client to speak your service's API (see Component 1: The M365 Gateway above). The three essential capabilities are session identity, message exchange, and authentication — the URL paths and payload shapes are flexible.

Component 3: The M365 App Package

The app package is a ZIP containing a manifest and icons that registers the gateway as a Custom Engine Agent in Microsoft 365.

Critical manifest fields

{ "bots": [{ "botId": "<BOT_APP_ID>", "scopes": ["personal"] }], "webApplicationInfo": { "id": "<BOT_SSO_APP_ID>", "resource": "api://botid-<BOT_APP_ID>" }, "copilotAgents": { "customEngineAgents": [{ "type": "bot", "id": "<BOT_APP_ID>", "functionsAs": "agentOnly" }] }, "validDomains": [ "<gateway-host>.azurecontainerapps.io", "token.botframework.com" ] }

Deployment Sequence

Follow this exact order. Each step depends on the previous outputs.

Step details:

Step	What	Depends on
1	Create service app + bot app in Entra ID	Tenant access
2	Grant admin consent for both delegated permission chains	Step 1
3	Create Azure Bot resource, configure OAuth connection (SERVICE_CONNECTION)	Steps 1-2
4	Provision downstream resources (databases, APIs, warehouses)	Independent
5	Build container image, deploy Agentic Service to ACA	Steps 1-4
6	Validate /healthz, session creation, first turn, OBO to downstream	Step 5
7	Build container image, deploy M365 Gateway to ACA	Steps 1-3, 5
8	Validate /healthz, gateway-to-service forwarding	Steps 6-7
9	Build manifest.json + icons into ZIP package	Steps 1, 7
10	Upload ZIP to Microsoft 365 admin center or Graph API	Step 9
11	Set Azure Bot messaging endpoint to https://<gateway>/api/messages	Steps 3, 7
12	Open M365 Copilot, send a message, verify end-to-end flow	Steps 10-11

Operatorized bootstrap variant

For a reusable customer/operator flow, this pattern works better when it is packaged into two main scripts plus two-layer env files:

Operator input env

– small, human-edited

– contains only the initial tenant/subscription/resource naming values and any demo-user identities

Generated runtime env

– script-owned

– stores discovered URLs, app IDs, secrets, package IDs, and image refs

Recommended split:

bootstrap-azure-demo.sh

– preflight Azure login and permissions

– create or reuse infra

– create/reuse app registrations

– enforce admin-consent success

– build and publish images

– deploy the service and the gateway

– wire Azure Bot and OAuth connection

bootstrap-m365-demo.sh

– build the Teams/M365 app package

– publish to the tenant catalog

– self-install for the signed-in operator

This keeps the operator surface area small while preserving advanced lower-level scripts for recovery or debugging.

Environment variables reference

Agentic Service

Variable	Example	Purpose
AZURE_TENANT_ID	b5d67878-...	Entra tenant (for OBO #2)
SERVICE_CLIENT_ID	<service-app-id>	Service app registration (for OBO #2)
SERVICE_CLIENT_SECRET	<secret>	Service OBO client credential
DOWNSTREAM_OBO_SCOPE	<resource-id>/.default	Downstream resource scope for OBO #2
AZURE_OPENAI_ENDPOINT	https://....cognitiveservices.azure.com	LLM endpoint
AZURE_OPENAI_DEPLOYMENT	gpt-4o	Model deployment name
DATABRICKS_WAREHOUSE_ID	warehouse-123	Optional fixed warehouse id if already known
DATABRICKS_AUTO_CREATE_WAREHOUSE	true	Allow bootstrap to create a warehouse when none exists

M365 Gateway

Variable	Example	Purpose
AZURE_TENANT_ID	b5d67878-...	Entra tenant
BOT_APP_ID	<bot-app-id>	Bot registration
BOT_APP_PASSWORD	<secret>	Bot client secret
SERVICE_BASE_URL	https://my-service.azurecontainerapps.io	Service endpoint
SERVICE_API_SCOPE	api://<service-app-id>/access_as_user	OBO #1 target scope
SERVICE_EXPECTED_AUDIENCE	api://<service-app-id>	JWT audience validation

Repeatability guidance

If you expect operators to rerun the deployment in the same working copy:

keep operator-owned inputs separate from generated runtime values
persist app IDs/object IDs after first create and prefer those on reruns
avoid re-identifying Entra apps by display name alone
bind the generated runtime env to a signature of the operator inputs so a new tenant, subscription, or prefix does not inherit stale values from an old run
prefer early preflight failures over deep runtime failures

Reference Implementation

A concrete implementation of this pattern is available here:

👉 dbx-mcp-copilot reference repo

This repository demonstrates the full end-to-end application of the architecture described in this guide, including:

A stateful agentic service deployed on Azure Container Apps
A thin M365 Gateway (wrapper) that exposes the service as a Custom Engine agent
Chained OBO flows enabling delegated access to downstream systems (e.g., Databricks)
Integration with the Microsoft Agent Framework for orchestration
Complete operator runbook, CI/CD pipelines, and app packaging for M365 deployment [github.com]

The current implementation is centered on a Daily Account Planner MVP, which illustrates how a real-world agent can:

Maintain session state and multi-turn reasoning
Access enterprise data using user-delegated tokens
Surface through Microsoft 365 Copilot without rewriting the agent into Copilot-native constructs [github.com]

Why this repo is useful

Concrete mapping of concepts → code
Every component in this guide (Gateway, Service, OBO chain) is implemented with working infrastructure and scripts.
Reusable gateway pattern
The M365 wrapper in the repo is intentionally thin and reusable — aligning with the pattern described in this document and designed to be adapted to other agentic services. [github.com]
Operator-ready deployment
Includes end-to-end bootstrap scripts, environment modeling, and CI/CD flows for repeatable deployments in real tenants.

GPT Capability in Understanding Coordinates: How GPT-5.4 Transforms Spatial Precision

jihyeseo — Thu, 23 Apr 2026 19:00:00 GMT

Why I Ran This Experiment

This work started not as a benchmarking exercise, but as a practical problem: I needed to automatically extract panel regions from PDF-format electrical Single-Line Diagram (SLD) drawings using OpenAI models . All experiments were conducted using OpenAI models in Microsoft Foundry- Microsoft's unified platform for building generative AI applications.

The downstream goal was a pipeline that combines GPT model with Azure Document Intelligence to generate Bills of Materials (BOMs) — a project I wrote about separately in Extracting BOMs from Electrical Drawings with AI: Azure OpenAI GPT-5 + Azure Document Intelligence Pipeline.

Before building that pipeline, I needed a clear-eyed answer to a deceptively simple question: how well can GPT actually understand and return pixel-level coordinates from an image? If the model can't reliably locate a panel bounding box, the rest of the pipeline doesn't matter.

When I first ran these tests against GPT-5.2, the results were mixed — good enough to be promising, but inconsistent enough to leave clear room for improvement. I tried many workarounds: feeding image dimensions explicitly, overlaying coordinate grids, enabling extended reasoning, and building iterative self-correction loops. Each helped, but required deliberate engineering effort. Then GPT-5.4 was released. Re-running the same benchmark revealed that most of those workarounds were no longer necessary.

Context: All experiments use a fixed CAD-style test image (847 × 783 px) with a known ground-truth bounding box at [135, 165, 687, 619]. Accuracy is measured by Intersection over Union (IoU) — a score of 1.0 is a perfect match. Every test was run 5 times and averaged.

Figure 1 — The clean electrical SLD drawing (847×783 px) used as the base test image
for all coordinate experiments.

The Experiment Design

I designed experiments across two axes: prompt strategy (how spatial information is presented to the model) and reasoning mode (standard vs. extended reasoning). Each combination was tested across both GPT-5.2 and GPT-5.4, producing 4 conditions per test. GPT-5.2 and GPT-5.4 were each tested under two reasoning modes (None vs. High), resulting in four conditions in total.

Single-Shot Strategies (Tests 1–5)

These tests have no iterative validation loop — the model gets one prompt and returns its answer. Each test was run 5 times and the results averaged, so the scores reflect consistency, not a single lucky attempt. The differences between tests lie in how spatial information is framed in the prompt.

Test 1 is a simple sanity check: can the model understand percentage-based coordinates at all? The model receives the clean image (no overlay) and is asked: "return the pixel coordinate at 30% width, 50% height." The expected answer is (254, 392).

GPT-5.2 gets the X coordinate roughly right (~254–260), but the Y coordinate scatters wildly — predictions range from 260 to 322, consistently 100+ pixels above the correct position. GPT-5.4 returns (254, 392) on every single run, essentially pixel-perfect.

Figure 2 — Test 1 results: GPT-5.2 (left) scatters predictions 100+ pixels above the expected point (red star). GPT-5.4 (right) nails it exactly on all 5 runs.

Even on this simple sanity check, the gap is stark: GPT-5.4 is pixel-perfect from the start, while GPT-5.2 shows a clear Y-axis bias. But a single-point test doesn't tell us how well the models handle real spatial tasks. The next question: can they detect a full bounding box?

Tests 2–5: Bounding Box Detection with Increasing Prompt Richness

Tests 2–5 move to the real task: detecting a bounding box drawn on the image. Each test sends a different version of the same base image, with progressively richer spatial context in the prompt:

Table 1 — Single-shot test descriptions: prompt strategy and input type for Tests 1–5.Figure 3 — Input images for Tests 2–5, from simplest (orange bbox only) to most structured (numbered grid).

Feedback Loop Strategies (Tests 6A–7B)

These tests add an iterative validation loop: the model's predicted bounding box is overlaid on the image and sent back for self-correction — up to 5 iterations (early stop at IoU ≥ 0.99). All feedback tests share the same two-phase structure: an init step (first prediction) and a validation loop (iterative correction).

All feedback tests use the same two images (init + validation overlay), but differ in prompt strategy and color assignment. Image-wise, they fall into two groups:

Group A — Orange GT (Tests 6A, 6C, 7A)

Figure 4a — Feedback loop input images. Orange GT box with blue prediction

Group B — Color Bias / Blue GT (Tests 6B, 6D, 7B)

Figure 4b — Feedback loop input images. Group B (bottom): colors swapped to test color-role priors.

What differs between tests in the same group: The images are identical, but the prompt changes. 6A/6B use holistic comparison ("compare and correct"). 6C/6D additionally send the full history of past predictions as multi-image input. 7A/7B ask for per-edge directional judgments ("move left/right/up/down/none" for each edge independently).

Results

1. Model version is the single biggest factor

Across every test, GPT-5.4 dramatically outperforms GPT-5.2. The gap is not incremental — it's the difference between a bounding box that roughly overlaps the target and one that is essentially pixel-perfect. GPT-5.4 achieved an IoU of 0.99 or above on its very first attempt on tests where GPT-5.2 had only scored between 0.76 and 0.88.

Figure 5 — Single-shot IoU across all 4 conditions.
GPT-5.4 (green bars) consistently hits ≥0.99 regardless of prompt strategy or reasoning mode.
GPT-5.2 (blue bars) ranges from 0.76 to 0.92.

2. GPT-5.2 is inconsistent; GPT-5.4 locks in

Raw averages only tell half the story. GPT-5.2 is unpredictable: on the exact same test with the exact same prompt and image, results fluctuate wildly between runs. The standard deviation on Test 2 is ±0.084 — meaning a single run could land anywhere from 0.66 to 0.88. GPT-5.4 stays within ±0.003.

The scatter plots below make this viscerally clear. Each dot is one API call — notice how GPT-5.2 dots spray across the IoU range while GPT-5.4 dots stack on top of each other:

Figure 6a — GPT-5.2: per-run IoU across Tests 2–5.
Wide scatter on simpler prompts (Test 2: 0.66–0.88); reasoning mode (orange) provides a lift that shrinks with richer prompts
(Δmean shown below each panel).Figure 6b — GPT-5.4: same view — all dots cluster at 0.97–1.0. No meaningful variance, no reasoning benefit.

Production implication: With GPT-5.2, you couldn't rely on a single inference call — building a reliable pipeline would require multiple calls and majority voting, multiplying latency and cost. With GPT-5.4, a single call is sufficient.

3. Reasoning mode reduced variance for GPT-5.2; GPT-5.4 didn't need it

For GPT-5.2, enabling extended reasoning (reasoning: high) provided a meaningful boost — especially when the prompt was sparse. On Test 2 (bare image, no spatial context), reasoning added +0.076 IoU and visibly tightened the spread of results across runs. As prompts got richer, the benefit shrank: with a grid overlay (Test 4), reasoning added only +0.007. In other words, reasoning mode acted as a compensating mechanism — filling in the gaps when the prompt alone didn't provide enough spatial scaffolding.

For GPT-5.4, reasoning mode offered no additional benefit on this class of task. The base model already achieves 0.99+ IoU, so there was simply no room for improvement. In a few cases the reasoning runs showed marginal regressions (−0.005 to −0.015), likely within noise. The takeaway isn't that reasoning mode is harmful in general, but rather that a spatial-coordinate task at this complexity level doesn't require it when the underlying model already has strong coordinate understanding.

Figure 7 — Effect of reasoning mode: GPT-5.2 gains +0.04–0.08 from reasoning (blue bars), largest on sparse prompts.

GPT-5.4 shows no meaningful gain (green bars near zero).

4. Richer prompts close the gap (but only for GPT-5.2)

For GPT-5.2, providing more spatial context in the prompt made a big difference: from 0.765 (Test 2, no info) to 0.910 (Test 4, grid overlay) — a +0.145 IoU gain just from adding visual reference rulers to the image. Telling the model the image dimensions (Test 3) was a "free win" that cost nothing.

For GPT-5.4, all prompt variants produce essentially the same result (0.989–0.997). The model already understands spatial coordinates well enough that extra scaffolding adds no value.

Figure 7 — Prompt information richness: GPT-5.2 climbs steeply as more spatial context is provided. GPT-5.4 is flat at ≥0.99 regardless.

If you're still on GPT-5.2: Always inject image dimensions into the prompt (free). Use grid overlays for the biggest single-shot gain (+0.145 IoU). With GPT-5.4, none of this is needed.

5. Validation loops: essential for GPT-5.2, Option for GPT-5.4

The feedback loop tests (6A–7B) showed that iterative self-correction genuinely helped GPT-5.2 improve from its initial prediction. For example, in Test 7A (directional feedback), GPT-5.2 improved from an init IoU of 0.926 to a best of 0.969 over 5 iterations.

For GPT-5.4, every single run hit IoU ≥ 0.99 on iteration 1 and early-stopped immediately. There was nothing left to correct. The validation loop infrastructure — overlay rendering, multi-turn prompting, iteration logic — becomes dead code you can remove from your pipeline.

Figure 8 — Validation loop effect (No Reasoning): GPT-5.2 init (light blue) improves to best (dark blue) over iterations.
GPT-5.4 (green) starts at ≥0.99 and early-stops at iteration 1.

6. Prompt instruction matters: holistic vs directional feedback

Comparing 6A/6B (holistic: "compare the two boxes and correct") with 7A/7B (directional: "for each edge, decide which direction to move"), the directional approach consistently reached higher best IoU for GPT-5.2. The per-edge structured output forced the model to reason about each boundary independently rather than making a holistic guess.

Separately, the color bias tests (6B, 7B — GT drawn in blue instead of orange) revealed that swapping GT/prediction colors drops the initial accuracy significantly. In 6A (orange GT) the init IoU was 0.937, but in 6B (blue GT) it dropped to 0.850. This suggests GPT models have learned color-role priors — orange is "expected" as the ground truth color.

However, the validation loop largely recovers this gap: after 5 iterations, 6A and 6B converge to similar best IoU (~0.96). The directional variants (7A, 7B) show the same pattern but converge faster.

Figure 9 — Color bias effect on GPT-5.2 (No Reasoning).
Left: initial accuracy drops when GT is drawn in blue.
Right: after the validation loop, the gap closes. Directional feedback (7A/7B) shows the same pattern.

For GPT-5.4: Color bias has no measurable effect. All variants (6A/6B/7A/7B) hit 0.994–0.998 IoU on iteration 1 regardless of color assignment.

Summary: What Changed from GPT-5.2 to GPT-5.4

The story of this benchmark is really about engineering workarounds that became unnecessary. Here's what we built for GPT-5.2 and whether you still need it:

Grid overlays & image dimensions in prompt — Gave +0.05–0.15 IoU for GPT-5.2. Not needed for GPT-5.4 (already ≥0.99 without it).
Extended reasoning mode — Gave +0.04–0.08 IoU for GPT-5.2. No benefit for GPT-5.4 on this task (already at ceiling without it).
Validation loops (iterative self-correction) — Improved GPT-5.2 by +0.02–0.10 IoU over 5 iterations. Unnecessary for GPT-5.4 (early-stops at iteration 1).
Multiple runs & voting — Required for GPT-5.2 due to ±0.08 variance. Not needed for GPT-5.4 (±0.003 variance, single call sufficient).
Color convention management — GPT-5.2 showed color bias (−0.09 IoU when colors swapped). No effect on GPT-5.4.

GPT-5.4 doesn't just perform better — it makes entire categories of pipeline engineering unnecessary. For clean, CAD-style images like the ones tested here, GPT-5.4 dramatically reduces prompt engineering overhead: grid overlays, image dimension injection, reasoning mode, and validation loops — all of which required deliberate effort with GPT-5.2 — are no longer necessary. This translates directly to simpler pipelines, lower latency, and lower cost. That said, for more complex scenarios — multiple overlapping panels, cluttered backgrounds, or ambiguous region boundaries — iterative validation loops could still prove valuable, and we plan to explore this in future work.

This benchmark started as a sanity check and turned into a clear signal: GPT-5.4 represents a genuine leap in spatial coordinate understanding, not just a marginal iteration. The gap between 0.765 and 0.997 IoU on an identical task is the difference between a prototype experiment and a production-ready component.

Try It Yourself

Ready to explore GPT-5.4's spatial precision capabilities? Here are ways to get started:

Sample notebooks for bounding box extraction test : github
Read the companion post: Extracting BOMs from Electrical Drawings with AI: Azure OpenAI GPT-5 + Azure Document Intelligence — See how this benchmark informed a production pipeline

Automate Prior Authorization with AI Agents - Now Available as a Foundry Template

amimukherjee — Thu, 23 Apr 2026 16:00:00 GMT

By Amit Mukherjee · Principal Solutions Engineer, Microsoft Health & Life Sciences

Lindsey Craft-Goins · Technology Leader - Cloud & AI Platforms, Health & Life Sciences

Joel Borellis · Director Solutions Engineering - Cloud & AI Platforms, Health & Life Sciences

Prior authorization (PA) is one of the most expensive bottlenecks in U.S. healthcare. Physicians complete an average of 39 PA requests per week, spending roughly 13 hours of physician-and-staff time on PA-related work (AMA 2024 Prior Authorization Physician Survey). Turnaround averages 5–14 business days, and PA alone accounts for an estimated $35 billion in annual administrative spending (Sahni et al., Health Affairs Scholar, 2024).

The regulatory clock is now ticking. CMS-0057-F mandates electronic PA with 72-hour urgent response starting in 2026. Forty-nine states plus DC already have PA laws on the books, and at least half of all U.S. state legislatures introduced new PA reform bills this year, including laws specifically targeting AI use in PA decisions (KFF Health News, April 2026).

Today we’re making the Prior Authorization Multi-Agent Solution Accelerator available as a Microsoft Foundry template. Health plan payers can deploy a working, four-agent PA review pipeline to Azure using the Azure Developer CLI (“azd”) with a single command in supported environments, then customize it to their policies, workflows, and EHR environment.

Try it now: Find the template in the Foundry template gallery, or clone directly from github.com/microsoft/Prior-Authorization-Multi-Agent-Solution-Accelerator

What the template delivers

The accelerator deploys four specialist Foundry hosted agents (Compliance, Clinical Reviewer, Coverage, and Synthesis), each independently containerized and managed by Foundry. In internal testing with synthetic demo cases, the pipeline reduced review workflow, from beginning to completion in under 5 minutes per case.

Agent	Role	Key capability
Compliance	Documentation check	10-item checklist with blocking/non-blocking flags
Clinical Reviewer	Clinical evidence	ICD-10 validation, PubMed + ClinicalTrials.gov search
Coverage	Policy matching	CMS NCD/LCD lookup, per-criterion MET/NOT_MET mapping
Synthesis	Decision rubric	3-gate APPROVE/PEND with weighted confidence scoring

Compliance and Clinical run in parallel. Coverage runs after clinical findings are ready. Synthesis evaluates all three outputs through a three-gate rubric. The result is a structured recommendation with per-criterion confidence scores and a full audit trail, not a black-box answer.

Solution architecture

The accelerator runs entirely on Azure. The frontend and backend deploy as Azure Container Apps. The four specialist agents are hosted by Microsoft Foundry. Real-time healthcare data flows through third-party MCP servers.

Figure 1: Azure solution architecture

How the pipeline works

The four agents execute in a structured parallel-then-sequential pipeline. Compliance and Clinical run simultaneously in Phase 1. Coverage runs after clinical findings are ready. The Synthesis agent applies a three-gate decision rubric over all prior outputs.

Figure 2: Agentic architecture, hosted agent pipeline

Compliance and Clinical run in parallel via asyncio.gather, since neither depends on the other. Coverage runs sequentially after Clinical because it needs the structured clinical profile for criterion mapping. Synthesis evaluates all three outputs through a three-gate rubric (Provider, Codes, Medical Necessity) with weighted confidence scoring: 40% coverage criteria + 30% clinical extraction + 20% compliance + 10% policy match. The total pipeline time is bound by the slowest parallel agent plus the sequential agents, not the sum. In internal testing with synthetic demo cases, this architecture indicated materially reduced processing time compared to sequential manual workflows.

Under the hood

For the architect in the room, here are four design decisions worth knowing about:

Foundry hosted agents: Each agent is independently containerized, versioned, and managed by Foundry’s runtime. The FastAPI backend is a pure HTTP dispatcher. All reasoning happens inside the agent containers, and there are no code changes between local (Docker Compose) and production (Foundry); the environment variable is the only switch.
Structured output: Every agent uses MAF’s response_format enforcement to produce typed Pydantic schemas at the token level. No JSON parsing, no malformed fences, no free-form text. The orchestrator receives typed Python objects; the frontend receives a stable API contract.
Keyless security: DefaultAzureCredential throughout, so no API keys are stored anywhere. Managed Identity handles production; azd tokens handle local development. Role assignments are provisioned automatically by Bicep at deploy time.
Observability: All agents emit OpenTelemetry traces to Azure Application Insights. The Foundry portal shows per-agent spans correlated by case ID. End-to-end latency, per-agent contribution, and error rates are visible from day one with no additional configuration.

For the full architecture documentation, agent specifications, Pydantic schemas, and extension guides, see the GitHub repository.

Why this matters now

Human-in-the-loop by design

The system runs in LENIENT mode by default: it produces only APPROVE or PEND and is not designed to produce automated DENY outcomes in its default configuration. Every recommendation requires a clinician to Accept or Override with documented rationale before the decision is finalized. Override records flow to the audit PDF, notification letters, and downstream systems. This directly addresses the emerging wave of state legislation governing AI use in PA decisions.

Domain experts own the rules

Agent behavior is defined in markdown skill files, not Python code. When CMS updates a coverage determination or a plan changes its commercial policy, a clinician or compliance officer edits a text file and redeploys. No engineering PR required.

Real-time healthcare data via MCP

Agents connect to five MCP servers for real-time data: ICD-10 codes, NPI Registry, CMS Coverage policies, PubMed, and ClinicalTrials.gov. This incorporates real‑time clinical reference data sources to inform agent recommendations.

Third-party MCP servers are included for demonstration with synthetic data only. Their inclusion does not constitute an endorsement by Microsoft. See the GitHub repository for production migration guidance.

Audit-ready from day one

Every case generates an 8-section audit justification PDF with per-criterion evidence, data source attribution, timestamps, and confidence breakdowns. Clinician overrides are recorded in Section 9. Notification letters (approval and pend) are generated automatically. These artifacts are designed to support CMS-0057-F documentation requirements.

Deploy in under 15 minutes

From the Foundry template gallery or from the command line:

git clone https://github.com/microsoft/Prior-Authorization-Multi-Agent-Solution-Accelerator

cd Prior-Authorization-Multi-Agent-Solution-Accelerator

azd up

That single command provisions Foundry, Azure Container Registry, Container Apps, builds all Docker images, registers the four agents, and runs health checks. The demo is live with a synthetic sample case as soon as deployment completes.

What’s included	What you customize
4 Foundry hosted agents	Payer-specific coverage policies
FastAPI orchestrator + Next.js frontend	EHR/FHIR integration for clinical notes
5 MCP healthcare data connections	Self-hosted MCP servers for production PHI
Audit PDF + notification letter generation	Authentication (Microsoft Entra ID)
Full Bicep infrastructure-as-code	Persistent storage (Cosmos DB / PostgreSQL)
OpenTelemetry + App Insights observability	Additional agents (Pharmacy, Financial)

Built on

Microsoft Foundry + Foundry hosted agents · Microsoft Agent Framework (MAF) · Azure OpenAI gpt-5.4 · Azure Container Apps · Azure Developer CLI + Bicep · OpenTelemetry + Azure Application Insights · DefaultAzureCredential (keyless, no secrets)

Full architecture documentation, agent specifications, and extension guides are in the GitHub repository.

Get started

Foundry template gallery: Search “AI-Powered Prior Authorization for Healthcare” in the Foundry template section
GitHub: github.com/microsoft/Prior-Authorization-Multi-Agent-Solution-Accelerator

Disclaimers

Not a medical device. This solution accelerator is not a medical device, is not FDA-cleared, and is not intended for autonomous clinical decision-making. All AI recommendations require qualified clinical review before any authorization decision is finalized.

Not production-ready software. This is an open-source reference architecture (MIT License), not a supported Microsoft product. Customers are solely responsible for testing, validation, regulatory compliance, security hardening, and production deployment.

Performance figures are illustrative. Metrics cited (including processing time reductions) are based on internal testing with synthetic demo data. Actual results will vary based on case complexity, infrastructure, and configuration.

Third-party services included for demonstration only; not endorsed by Microsoft. Customers should evaluate providers against their compliance and data residency requirements.

The demo uses synthetic data only. Customers deploying real patient data are responsible for HIPAA compliance and establishing appropriate Business Associate Agreements.

This accelerator is intended to help customers align documentation workflows with CMS‑0057‑F requirements but has not been independently validated or certified for regulatory compliance.

Failed to add tool to agent - Preview Feature Required?

ArneVG — Thu, 23 Apr 2026 15:09:40 GMT

Hi,

We’ve recently run into an issue where we’re no longer able to add tools to our Foundry agent. This was previously working without problems in our development environment, but now every attempt results in the following error:

“Failed to add tool to agent Request failed with status code 403.”

After inspecting the request in the browser’s developer console, we noticed an additional message:

"This operation requires the following opt-in preview feature(s): AgentEndpoints=V1Preview. Include the 'Foundry-Features: AgentEndpoints=V1Preview' header in your request."

How can we opt in for this foundry preview feature? and when was this change introduced?
We are unsure if the issue is related the the preview feature missing, or some other forbidden issue.

Any help would be very much appreciated.

Kind regards,

Arne

Three tiers of Agentic AI - and when to use none of them

sgangaramani — Wed, 22 Apr 2026 16:00:00 GMT

Every enterprise has an AI agent. Almost none of them work in production.

Walk into any enterprise technology review right now and you will find the same thing. Pilots running. Demos recorded. Steering committees impressed. And somewhere in the background, a quiet acknowledgment that the thing does not actually work at scale yet.

OutSystems surveyed nearly 1,900 global IT leaders and found that 96% of organizations are already running AI agents in some capacity. Yet only one in nine has those agents operating in production at scale. The experiments are everywhere. The production systems are not.

That gap is not a capability problem. The infrastructure has matured. Tool calling is standard across all major models. Frameworks like LangGraph, CrewAI, and Microsoft Agent Framework abstract orchestration logic. Model Context Protocol standardizes how agents access external tools and data sources. Google's Agent-to-Agent protocol now under Linux Foundation governance with over 50 enterprise technology partners including Salesforce, SAP, ServiceNow, and Workday standardizes how agents coordinate with each other. The protocols are in place. The frameworks are production ready.

The gap is a selection and governance problem.

Teams are building agents on problems that do not need them. Choosing the wrong tier for the ones that do. And treating governance as a compliance checkbox to add after launch, rather than an architectural input to design in from the start. The same OutSystems research found that 94% of organizations are concerned that AI sprawl is increasing complexity, technical debt, and security risk and only 12% have a centralized approach to managing it. Teams are deploying agents the way shadow IT spread through enterprises a decade ago: fast, fragmented, and without a shared definition of what production-ready actually means.

I've built agentic systems across enterprise clients in logistics, retail, and B2B services. The failures I keep seeing are not technology failures. They are architecture and judgment failures problems that existed before the first line of code was written, in the conversation where nobody asked the prior question.

This article is the framework I use before any platform conversation starts.

What has genuinely shifted in the agentic landscape

Three changes are shaping how enterprise agent architecture should be designed today and they are not incremental improvements on what existed before.

The first is the move from single agents to multi-agent systems. Databricks' State of AI Agents report drawing on data from over 20,000 organizations, including more than 60% of the Fortune 500 found that multi-agent workflows on their platform grew 327% in just four months. This is not experimentation. It is production architecture shifting. A single agent handling everything routing, retrieval, reasoning, execution is being replaced by specialized agents coordinating through defined interfaces. A financial organization, for example, might run separate agents for intent classification, document retrieval, and compliance checking each narrow in scope, each connected to the next through a standardized protocol rather than tightly coupled code.

The second is protocol standardization. MCP handles vertical connectivity how agents access tools, data sources, and APIs through a typed manifest and standardized invocation pattern. A2A handles horizontal connectivity how agents discover peer agents, delegate subtasks, and coordinate workflows. Production systems today use both. The practical consequence is that multi-agent architectures can be composed and governed as a platform rather than managed as a collection of one-off integrations.

The third is governance as the differentiating factor between teams that ship and teams that stall. Databricks found that companies using AI governance tools get over 12 times more AI projects into production compared to those without. The teams running production agents are not running more sophisticated models. They built evaluation pipelines, audit trails, and human oversight gates before scaling not after the first incident.

Tier 1 - Low-code agents: fast delivery with a defined ceiling

The low-code tier is more capable than it was eighteen months ago. Copilot Studio, Salesforce Agentforce, and equivalent platforms now support richer connector libraries, better generative orchestration, and more flexible topic models. The ceiling is higher than it was. It is still a ceiling.

The core pattern remains: a visual topic model drives a platform-managed LLM that classifies intent and routes to named execution branches. Connectors abstract credential management and API surface. A business team — analyst, citizen developer, IT operations — can build, deploy, and iterate without engineering involvement on every change. For bounded conversational problems, this is the fastest path from requirement to production.

The production reality is documented clearly. Gartner data found that only 5% of Copilot Studio pilots moved to larger-scale deployment. A European telecom with dedicated IT resources and a full Microsoft enterprise agreement spent six months and did not deliver a single production agent. The visual builder works. The path from prototype to production, production-grade integrations, error handling, compliance logging, exception routing is where most enterprises get stuck, because it requires Power Platform expertise that most business teams do not have.

The platform ceiling shows up predictably at four points. Async processing anything beyond a synchronous connector call, including approval chains, document pipelines, or batch operations cannot be handled natively. Full payload audit logs platform logs give conversation transcripts and connector summaries, not structured records of every API call and its parameters. Production volume concurrency limits and message throughput budgets bind faster than planning assumptions suggest. Root cause analysis in production you cannot inspect the LLM's confidence score or the alternatives it considered, which makes diagnosing misbehavior significantly harder than it should be.

The correct diagnostic: can this use case be owned end-to-end by a business team, covered by standard connectors, with no latency SLA below three seconds and no payload-level compliance requirement? Yes, low code is the correct tier. Not a compromise. If no on any point, continue.

If low-code is the right call for your use case: Copilot Studio quickstart

Tier 2 - Pro-code agents: the architecture the current landscape demands

The defining pattern in production pro-code architecture today is multi-agent. Specialized agents per domain, coordinating through MCP for tool access and A2A for peer-to-peer delegation, with a governance layer spanning the entire system.

What this looks like in practice: a financial organization handling incoming compliance queries runs separate agents for intent classification, document retrieval, and the compliance check itself. None of these agents tries to do all three jobs. Each has a narrow responsibility, a defined input/output contract typed against a JSON Schema, and a clear handoff boundary. The 327% growth in multi-agent workflows reflects production teams discovering that the failure modes of monolithic agents topic collision, context overflow, degraded classification as scope expands are solved by specialization, not by making a single agent more capable.

The discipline that makes multi-agent systems reliable is identical to what makes single-agent systems reliable, just enforced across more boundaries: the LLM layer reasons and coordinates; deterministic tool functions enforce. In a compliance pipeline, no LLM decides whether a document satisfies a regulatory requirement. That evaluation runs in a deterministic tool with a versioned rule set, testable outputs, and an immutable audit log. The LLM orchestrates the sequence. The tool produces the compliance record. Mixing these letting an LLM evaluate whether a rule pass collapses the audit trail and introduces probabilistic outputs on questions that have regulatory answers.

MCP is the tool interface standard today. An MCP server exposes a typed manifest any compliant agent runtime can discover at startup. Tools are versioned, independently deployable, and reusable across agents without bespoke integration code. A2A extends this horizontally: agents advertise capability cards, discover peers, and delegate subtasks through a standardised protocol. The practical consequence is that multi-agent systems built on both protocols can be composed and governed as a platform rather than managed as a collection of one-off integrations.

Observability is the architectural element that separates teams shipping production agents from teams perpetually in pilot. Build evaluation pipelines, distributed traces across all agent boundaries, and human review gates before scaling. The teams that add these after the first production incident spend months retrofitting what should have been designed in.

If pro-code is the right call for your use case: Foundry Agent Service

The hybrid pattern: still where production deployments land

The shift to multi-agent architecture does not change the hybrid pattern it deepens it. Low-code at the conversational surface, pro-code multi-agent systems behind it, with a governance layer spanning both.

On a logistics client engagement, the brief was a sales assistant for account managers shipment status, account health, and competitive context inside Teams. The business team wanted everything in Copilot Studio. Engineering wanted a custom agent runtime. Both were wrong.

What we built: Copilot Studio handled all high-frequency, low-complexity queries shipment tracking, account status, open cases through Power Platform connectors. Zero custom code. That covered roughly 78% of actual interaction volume. Requests requiring multi-source reasoning competitive positioning on a specific lane, churn risk across an account portfolio, contract renewal analysis delegated via authenticated HTTP action to a pro-code multi-agent service on Azure. A retrieval agent pulled deal history and market intelligence through MCP-exposed tools. A synthesis agent composed the recommendation with confidence scoring. Structured JSON back to the low-code layer, rendered as an adaptive card in Teams.

The HITL gate was non-negotiable and designed before deployment, not added after the first incident. No output reached a customer without a manager approval step. The agent drafts. A human sends.

This boundary low-code for conversational volume, pro-code for reasoning depth maps directly to what the research shows separates teams that ship from teams that stall. The organizations running agents in production drew the line correctly between what the platform can own and what engineering needs to own. Then they built governance into both sides before scaling.

The four gates - the prior question that still gets skipped

Run every candidate use case through these four checks before the platform conversation begins. None of the recent infrastructure improvements change what they are checking, because none of them change the fundamental cost structure of agentic reasoning.

Gate 1 - is the logic fully deterministic? If every valid output for every valid input can be enumerated in unit tests, the problem does not need an LLM. A rules engine executes in microseconds at zero inference cost and cannot produce a plausible-but-wrong answer. NeuBird AI's production ops agents which have resolved over a million alerts and saved enterprises over $2 million in engineering hours work because alert triage logic that can be expressed as rules runs in deterministic code, and the LLM only handles cases where pattern-matching is insufficient. That boundary is not incidental to the system's reliability. It is the reason for it.

Gate 2 - is zero hallucination tolerance required? With over 80% of databases now being built by AI agents per Databricks' State of AI Agents report the surface area for hallucination-induced data errors has grown significantly. In domains where a wrong answer is a compliance event financial calculation, medical logic, regulatory determinations irreducible LLM output uncertainty is disqualifying regardless of model version or prompt engineering effort. Exit to deterministic code or classical ML with bounded output spaces.

Gate 3 - is a sub-100ms latency SLA required? LLM inference is faster than it was eighteen months ago. It is not fast enough for payment transaction processing, real-time fraud scoring, or live inventory management. A three-agent system with MCP tool calls has a P50 latency measured in seconds. These problems need purpose-built transactional architecture.

Gate 4 - is regulatory explainability required? A2A enables complex agent coordination and delegation. It does not make LLM reasoning reproducible in a regulatory sense. Temperature above zero means the same input produces different outputs across invocations. Regulators in financial services, healthcare, and consumer credit require deterministic, auditable decision rationale. Exit to deterministic workflow with structured audit logging at every

Five production failure modes - one of them new

The four original anti-patterns are still showing up in production. A fifth has been added by scale.

Routing data retrieval through a reasoning loop. A direct API call returns account status in under 10ms. Routing the same request through an LLM reasoning step adds hundreds of milliseconds, consumes tokens on every call, and introduces output parsing on data that is already structured. The agent calls a structured tool. The tool calls the API. The agent never acts as the integration layer.

Encoding business rules in prompts. Rules expressed in prompt text drift as models update. They produce probabilistic output across invocations and fail in ways that are difficult to reproduce and diagnose. A rule that must evaluate correctly every time belongs in a deterministic tool function unit-tested, version-controlled, independently deployable via MCP.

No approval gate on CRUD operations. CRUD operations without a human approval step will eventually misfire on the input that testing did not cover. The gate needs to be designed before deployment, not added after the first incident involving a financial posting, a customer-facing communication, or a data deletion.

Monolithic agent for all domains. A single agent accumulating every domain leads predictably to topic collision, context overflow, and maintenance that becomes impossible as scope expands. Specialized agents per domain, coordinating through A2A, is the architecture that scales.

Ungoverned agent sprawl. This is the new one and currently the most prevalent. OutSystems found 94% of organizations concerned about it, with only 12% having a centralized response. Teams building agents independently across fragmented stacks, without shared governance, evaluation standards, or audit infrastructure, produce exactly the same organizational debt that shadow IT created but with higher stakes, because these systems make autonomous decisions rather than just storing and retrieving data. The fix is treating governance as an architectural input before deployment, not a compliance requirement after something breaks.

The infrastructure is ready. The judgment is not.

The tier decision sequence has not changed. Does the problem need natural language understanding or dynamic generation? No — deterministic system, stop. Can a business team own it through standard connectors with no sub-3-second latency SLA and no payload-level compliance requirement? Yes — low-code. Does it need custom orchestration, multi-agent coordination, or audit-grade observability? Yes — pro-code with MCP and A2A. Does it need both a conversational surface and deep backend reasoning? Hybrid, with a governance layer spanning both.

What has changed is that governance is no longer optional infrastructure to add when you have time. The data is unambiguous. Companies with governance tools get over 12 times more AI projects into production than those without. Evaluation pipelines, distributed tracing across agent boundaries, human oversight gates, and centralised agent lifecycle management are not overhead. They are what converts experiments into production systems. The teams still stuck in pilot are not stuck because the technology failed them. They are stuck because they skipped this layer.

The protocols are standardised. The frameworks are mature. The infrastructure exists. None of that is what is holding most enterprise agent programmes back.

What is holding them back is a selection problem disguised as a technology problem — teams building agents before asking whether agents are warranted, choosing platforms before running the four gates, and treating governance as a checkpoint rather than an architectural input.

I have built agents that should have been workflow engines. Not because the technology was wrong, but because nobody stopped early enough to ask whether it was necessary. The four gates in this article exist because I learned those lessons at clients' expense, not mine. The most useful thing I can offer any team starting an agentic AI project is not a framework selection guide. It is permission to say no — and a clear basis for saying it. Take the four gates framework to your next architecture review. If you have already shipped agents to production, I would like to hear what worked and what did not - comment below

What to do next

Three concrete steps depending on where you are right now.

If you have pilots that have not reached production: Run them through the four gates in this article before the next sprint. Gate 1 alone will eliminate a meaningful percentage of them. The ones that survive all four are your real candidates for production investment. Download the attached file for gated checklist and take it into your next architecture review.

If you are starting a new agent project: Do not open a platform before you have answered the gate questions. Once you have confirmed an agent is warranted and identified the tier, start here: Copilot Studio guided setup for low-code scenarios, or Foundry Agent Service for pro-code patterns with MCP and multi-agent coordination built in. Build governance infrastructure - evaluation pipeline, distributed tracing, HITL gates - before you scale, not after.

If you have already shipped agents to production: Share what worked and what did not in the Azure AI Tech Community — tag posts with #AgentArchitecture. The most useful signal for teams still in pilot is hearing from practitioners who have been through production, not vendors describing what production should look like.

References

OutSystems — State of AI Development Report - https://www.outsystems.com/1/state-ai-development-report

Databricks — State of AI Agents Report - https://www.databricks.com/resources/ebook/state-of-ai-agents

Gartner — 2025 Microsoft 365 and Copilot Survey - https://www.gartner.com/en/documents/6548002 (Paywalled primary source — publicly reported via techpartner.news: https://www.techpartner.news/news/gartner-microsoft-copilot-hype-offset-by-roi-and-readiness-realities-618118)

Anthropic — Model Context Protocol (MCP) - https://modelcontextprotocol.io

Google Cloud — Agent-to-Agent Protocol (A2A) . https://developers.googleblog.com/en/a2a-a-new-era-of-agent-interoperability

NeuBird AI — Production Operations Deployment Announcement NeuBird AI Closes $19.3M Funding Round to Scale Agentic AI Across Enterprise Production Operations

ReAct: Synergizing Reasoning and Acting in Language Models — Yao et al. https://arxiv.org/abs/2210.03629

Enterprise Integration Patterns — Gregor Hohpe & Bobby Woolf, Addison-Wesley https://www.enterpriseintegrationpatterns.com

Introducing Kimi K2.6 in Microsoft Foundry

RashaudSavage — Wed, 22 Apr 2026 15:56:21 GMT

We’re excited to welcome Moonshot AI’s Kimi K2.6 to Microsoft Foundry expanding the platform’s growing catalog of open and frontier models designed for real-world, production-grade AI systems.

Kimi K2.6 represents a new class of agentic, multimodal models built for long-horizon reasoning, coding, and autonomous execution—bringing developers closer to fully self-directed AI systems that can plan, act, and deliver outcomes end-to-end.

Why Kimi K2.6 matters

According to Moonshot AI, K2.6 is a native multimodal agentic model that advances capabilities in long-horizon coding, autonomous execution, and multi-agent orchestration.

This means developers can go beyond prompts and build systems where AI:

Plans and executes multi-step workflows
Writes, debugs, and refactors large codebases
Generates full applications—from UI to backend
Orchestrates multiple sub-agents to solve complex problems

What differentiates Kimi K2.6 is its focus on agentic intelligence at scale.

Unlike traditional models optimized for single responses, K2.6 is designed to:

Handle long-running tasks across hundreds of steps
Coordinate parallel sub-agents (“agent swarms”)
Combine reasoning with tool use and execution
Deliver complete outputs—documents, apps, workflows—in a single run

This aligns with the broader industry shift toward AI agents that operate more like systems than tools.

Built for developers: Coding, reasoning, and beyond

Kimi K2.6 builds on the Kimi K2 family, which introduced large-scale Mixture-of-Experts (MoE) architectures with up to 1 trillion parameters, optimized for reasoning, coding, and agent workflows.

With K2.6, those capabilities are extended further:

Deeper reasoning and planning for complex, multi-file coding tasks
Improved agent orchestration, enabling cleaner task decomposition
Stronger tool-use reliability across multi-step workflows
Multimodal inputs, combining text and visual understanding

The result is a model that is particularly well-suited for:

Developer copilots and coding agents
Document and knowledge workflows
Autonomous research and analysis pipelines
End-to-end application generation

Open models meet enterprise-grade infrastructure

Kimi K2.6 is part of a growing trend toward open, high-performance models that give developers flexibility without sacrificing capability.

In Microsoft Foundry, you can combine this openness with enterprise-grade features:

Unified API and SDKs across models
Model evaluation and observability tools
Built-in safety and governance controls
Flexible deployment options (global, regional, data zones)
Integration with agent frameworks and orchestration tools

This means you can experiment with Kimi K2.6 and seamlessly move to production—without re-architecting your stack.

Pricing

Token Type	Price per 1M tokens
Input tokens	$0.95
Output tokens	$4

Getting started

Kimi K2.6 is now available in Microsoft Foundry.

You can:

Explore the model in the Foundry catalog
Benchmark it against other models using built-in evaluations
Integrate it into your applications using the Foundry SDK

Extracting BOMs from Electrical Drawings with AI: Azure OpenAI GPT-5.4 + Azure Document Intelligence

jihyeseo — Wed, 22 Apr 2026 13:00:00 GMT

The Problem

Electrical engineering drawings — especially single-line diagrams (SLDs) — are notoriously hard to parse programmatically. They combine dense symbols, small text, and complex geometric structures, all varying in style across documents and vendors.

Figure 1. A typical electrical single-line diagram. A single page contains multiple panel regions (HV, TR, LV, GCP), each with dozens of electrical components, connection lines, and specification text — all of which must be correctly identified and assigned to the right panel for BOM extraction.

Traditionally, extracting a Bill of Materials (BOM) from these drawings has been a manual task — engineers would read through each diagram page by page and transcribe component lists by hand. It's time-consuming, error-prone, and doesn't scale.

The obvious question: can a Vision Language Model automate this? We had to rely primarily on the visual content of PDF-converted images alone — without guaranteed access to CAD vector data or metadata. That constraint shaped every technique described in this post.

Warning: Why naive inference fails: Feeding a full diagram page to a vision model and asking for a BOM fails catastrophically — too much visual complexity in a single context. Components get missed, hallucinated, or assigned to the wrong panel.

This post shares the practical techniques we discovered while building a pipeline to solve this problem. The methods are general — applicable to any task that requires extracting structured information from visually complex technical documents.

Pipeline Architecture: Divide and Conquer

The core insight: the unit of analysis must be the panel, not the page.

A single diagram page can contain dozens of panels, each representing a distinct electrical cabinet with its own components. Asking a model to extract a complete BOM from an entire page is asking it to simultaneously locate, read, and count every component across all panels — a task that proved too complex even for GPT-5.4.

The solution was to first identify and crop each panel as an independent region, then extract the BOM panel by panel. This transformed an intractable whole-page problem into a series of manageable, well-scoped sub-tasks.

Figure 2. Five-stage pipeline architecture. Each stage is color-coded by its primary tool: rule-based (gray), Azure Document Intelligence (orange), hybrid Azure OpenAI+Document Intelligence (purple), and Azure OpenAI GPT-5 vision (blue). Tool tags on the right indicate specific components used at each stage.

Technique 1: Azure Document Intelligence-First Detection with Targeted Azure OpenAI Supplement

The first challenge is identifying all figure regions on each page — the panel diagrams that contain the actual electrical components. SLD pages typically mix diagrams with title blocks, revision tables, and legend boxes (often along the right edge or bottom). All of these must be located before we can isolate the panels.

The key design decision: use Azure Document Intelligence (DI) as the primary detector — and reserve GPT-5.4 only for gaps that DI misses. DI's prebuilt-layout model is fast, deterministic, and cheap compared to an Azure OpenAI vision call. By maximizing DI coverage first, we minimize the number of expensive Azure OpenAI invocations needed to achieve full detection.

Two-Pass Document Intelligence Layout detection: Catching Occluded Regions

A single Azure Document Intelligence (DI) pass often misses figures that are visually occluded by larger detected regions — particularly smaller panels nested within or adjacent to large ones, and tables along the page edges. The solution: white-fill detected regions and re-run DI to reveal what was hidden underneath.

# Pass 1: Detect figures on original page
pass1 = analyze_page(di_client, "prebuilt-layout", image_path)
pass1_bboxes = [f["bbox"] for f in pass1["figures"]]

# Pass 2: White-fill Pass 1 regions → re-detect
if pass1_bboxes:
    white_fill_regions(image_path, pass1_bboxes, whitefill_path)
    pass2 = analyze_page(di_client, "prebuilt-layout", whitefill_path)

    # Merge & deduplicate by IoU
    all_figures = pass1["figures"] + pass2["figures"]
    deduped = _dedup_figure_bboxes(all_figures, iou_threshold=0.5)

This two-pass approach is especially effective at catching tables and annotation blocks along the right edge or bottom of the page that DI initially subsumes into a single large region.

Azure OpenAI GPT-5.4 Only for the Remaining Gaps

AAfter two Azure Document Intelligence(DI) passes, most figure regions are covered. For any remaining gaps, Azure OpenAI GPT-5.4 is called once with the DI-detected regions marked in purple on the image. The model only needs to identify unmarked areas — a much simpler task than full-page detection from scratch.

Key finding: DI detection is ~10× faster and significantly cheaper per call than an Azure OpenAI GPT-5.4 vision request. By using DI as the primary detector and Azure OpenAI only for supplemental gap-filling, the pipeline achieves comprehensive coverage while keeping cost and latency low. The two-pass technique further reduces Azure OpenAI's burden by maximizing what DI can find on its own.

Technique 2: Hybrid Azure OpenAI GPT + Document Intelligence for Text Localization

To segment panel areas, we first need to know where panel names appear in the image. Panel names act as anchor points — once we know their locations, we can use them as seeds to identify the boundaries of each panel region.

Neither GPT-5.4 nor Azure Document Intelligence alone is sufficient:

GPT-5.4 : identifies what the text says, but imprecise on exact pixel locations
Azure Document Intelligence: precise coordinates for all text, but struggles with domain-specific abbreviations

The solution: run both in parallel and cross-match results.

Figure 3. Two parallel tracks converge into cross-matching. Top: page is split into overlapping tiles → Azure OpenAI GPT-5.4 extracts panel name candidates per tile → aggregated and deduplicated. Bottom: Azure DI extracts all text with bounding boxes → rule + Azure OpenAI GPT-5.4 filters by type → cross-matching prioritizes rule-based alignment, with Azure OpenAI GPT-5.4 resolving unmatched cases.

Tile-Based Name Extraction

Rather than feeding the entire page to Azure OpenAI, we split it into overlapping vertical tiles (2000px wide, 400px overlap) and extract panel name candidates from each tile independently. This reduces visual complexity per call and improves recall.

# Split page into overlapping tiles for LLM name extraction
tiles = tile_page(image_path, tile_width=2000, overlap=400)

# Extract names from each tile independently
for tile_img, tile_coords in tiles:
    names = extract_names_from_tile(tile_img, llm_client, deployment)
    # LLM identifies SHORT CODES: "HV 1", "TR-2", "LV 3A"
    # Rejects: component labels, wire tags, terminal IDs

Figure 4. Effect of tile-based extraction vs. full-page extraction. Overlapping tiles reduce visual complexity per LLM call, improving panel name recall — especially for names near page edges.

Hallucination Guard

Azure OpenAI GPT-5.4 model sometimes fabricate panel names that don't exist in the image. We cross-validate all Azure OpenAI GPT-5.4 model-extracted names against the Azure Document Intelligence OCR text pool using fuzzy matching:

def hallucination_guard(names, di_lines):
    verified = []
    for name in names:
        if _name_in_ocr(name, ocr_texts):  # 3 matching modes:
            verified.append(name)           #   1. Exact substring
            # 2. Same length, ≤1 char diff
            # 3. Alphanumeric stripping (ignore spaces/punctuation)
        else:
            print(f"  Dropped '{name}' — no OCR match")
    return verified

6-Pass Rule Matching Engine

Once names are verified, we locate their exact pixel positions via a cascading rule engine that matches panel names against Azure Document Intelligence OCR bounding boxes with decreasing confidence:

def rule_match_panel_name(panel_name, di_lines, max_merge=3):
    # Pass 1: Exact match (case-insensitive)          → confidence 1.0
    # Pass 2: Alphanumeric (O/0, I/1 tolerance)       → confidence 0.9
    # Pass 3: Substring containment (≥3 chars)         → confidence 0.75
    # Pass 4: Multi-line merge (adjacent DI lines)     → confidence 0.7
    # Pass 5: OCR-confusable (1-char diff, v↔y, s↔5)  → confidence 0.6

    # Conflict resolution: shared bbox → keep highest confidence
    # LLM fallback: for still-unmatched names

Key finding: GPT-5.4 identifies what the panel names are (semantic), while Azure Document Intelligence provides where they are (geometric). The rule engine bridges the two with OCR-aware fuzzy matching. This hybrid approach is significantly more robust than either system alone.

Technique 3: Iterative Locate → Verify with Oscillation Guard

With panel names located, the next challenge is identifying the full panel boundary. This is the hardest stage: panels can be irregularly shaped, share edges with neighbors, and have boundaries formed by a mix of dashed, solid, and implied lines.

Rather than asking Azure OpenAI GPT-5.4 Model to find the boundary in one shot (which is unreliable), we implemented an iterative Locate → Verify correction loop with up to 10 attempts per panel.

Visual Prompt Composition

Each iteration constructs a carefully composed image for Azure OpenAI GPT-5.4, providing spatial context through color-coded overlays:

Locate Input

Blue box — target panel name (NAME:{panel_name})
Green boxes — other panels as spatial reference

Verify Input

Orange box — proposed panel bbox
Blue box — panel name location
Green boxes — neighboring panels

The verify step analyzes each of the four edges independently:

// Verify response — per-edge analysis
{
  "valid": false,
  "edges": {
    "x1": { "status": "expand", "delta": -45, "corrected": 120 },
    "y1": { "status": "ok" },
    "x2": { "status": "shrink", "delta": -30, "corrected": 850 },
    "y2": { "status": "expand", "delta": 60, "corrected": 1200 }
  },
  "corrected_bbox": [120, 200, 850, 1200]
}

Oscillation Detection

A critical failure mode: Azure OpenAI GPT-5.4 oscillates on an axis — expanding, then shrinking, then expanding — never converging. We detect this using a 3-value history per axis:

# Track last 3 corrections per axis
axis_history = {axis: [] for axis in ["x1", "y1", "x2", "y2"]}

for attempt in range(1, max_tries + 1):
    # ... locate and verify ...
    for axis in AXES:
        hist = axis_history[axis]
        if len(hist) >= 3:
            a, b, c = hist[-3:]
            # Detect: (a < b > c) or (a > b < c) → oscillating
            if (a < b and b > c) or (a > b and b < c):
                corrected[axis] = hist[-2]  # Freeze at previous value

The loop processes all panels on a page in batch mode — a single locate-all call positions all panels simultaneously, reducing per-panel LLM calls from N to 1.

Technique 4: Few-Shot Visual Prompting

Text prompts alone struggle to convey spatial concepts like "what a panel boundary looks like." The visual vocabulary of electrical drawings is too domain-specific to describe in words. The solution: provide GPT-5.4 with few-shot reference images directly in the prompt.

def locate_and_verify_batch(..., guide_image_paths=None):
    guides = list(guide_image_paths or [])
    # Prepend guide images before the actual input:
    loc_raw = call_llm(
        llm_client, deployment, locate_prompt,
        image_paths=guides + [locate_img_path],  # Guides first
    )

Figure 5. Few-shot reference for panel boundary recognition. Example 1: dashed rectangle enclosure. Example 2: mixed boundary with dashed lines, solid edge, and gap + vertical line as shared boundary. Example 3: non-rectangular region returns the full outer bounding box.

The benefits:

Reduces prompt length — no need to describe visual concepts in words
Improves consistency — the model interprets boundary types correctly across varying layouts
User-customizable — swapping in new guide images adapts to new drawing styles without code changes

Technique 5: Reasoning Mode — Performance vs. Cost

Azure OpenAI GPT-5.4's reasoning capability (reasoning={"effort": "low|medium|high"}) significantly affects both accuracy and latency. We ran systematic experiments across all four reasoning levels in two key pipeline stages: image area detection and BOM extraction.

Reasoning in Image Area Detection

Each reasoning level was tested 3 times on the region detection stage (Azure OpenAI GPT-5.4 detects and verifies figure boundaries).

Detection Quality

Figure 6. Region detection output across reasoning levels (High, Medium, Low, None × 3 runs). All levels produced comparable quality. high converged in fewer iterations (2), while medium/low sometimes needed 3.

Verification Iterations

Figure 7. Mean verification iterations per page (3 runs avg). high: 2.0, medium: 2.7, low: 3.0, none: 2.0. Lower reasoning needs slightly more iterations but converges to the same quality.

Processing Time

Figure 8. Mean LLM time for detection + verification (3 runs avg). high/medium ~170s, low ~52s, none ~15s.

Key finding: All reasoning levels (low, medium, high) produced similar quality and noticeably better results than none, but with increasing latency (~3× from low to high). Since there was no meaningful quality difference among reasoning levels, we chose low as the optimal setting — getting the benefit of reasoning at minimal latency cost.

Reasoning in BOM Extraction

For BOM extraction — reading component lists from cropped panel images — reasoning has a more pronounced accuracy impact:

Figure 9. Time vs. Accuracy: Low (~86%, ~2,300s) vs. Medium (~89–91%, ~3,900s) reasoning across 3 runs each. High (Timeout)

Low: ~85% Accuray, ~2,200–2,400s
Medium: ~85% ~91% Accuracy, ~3700-4100s
High :Timeout

Recommended Configurations

Pipeline Stage	Recommended	Rationale
Image region detection	`low`	Same quality at ~3× less cost
Region verification	`low`	Sufficient with rich visual context (color-coded overlays)
BOM extraction	`medium`	+3–5% accuracy; high causes timeouts

Key insight: Different stages need different reasoning levels. Use low for spatially-grounded tasks with rich visual context, medium for semantically-demanding reading tasks. Optimize inputs before scaling compute.

Technique 6: SVG Vector Boundary Detection

While the main pipeline relies on Azure OpenAI GPT-5.4's vision for panel boundary detection, we also explored a purely geometric approach that bypasses the GPT-5.4 model entirely — useful when CAD files can be exported as SVG vectors rather than raster images.

The core idea: panel boundaries are closed geometric shapes formed by line segments in the vector data. If we can extract meaningful line clusters and find closed cycles in the resulting graph, we can identify panel regions without any LLM calls.

Figure 10. Panel regions detected via SVG vector analysis — each color represents a distinct panel boundary identified through line clustering and cycle extraction, with no LLM involvement.

The Problem: Too Many Lines

A single SVG page can contain 17,000+ line segments — horizontal, vertical, and diagonal — mixing panel borders, component symbols, text strokes, and wiring lines. Attempting to work with raw segments directly is impractical.

Figure 11. Raw SVG line extraction from a single page: 15,742 horizontal + 1,106 vertical + 227 diagonal segments. The length distribution (right) shows most segments are very short (symbol strokes, text) — boundary lines are a tiny fraction of the total.

Chain-Based Line Clustering

The solution is clustering collinear, nearby segments into coherent boundary lines. We first scan fragments in order of position. If the next fragment is close enough, extend the current group. If the gap is too large, start a new one. Discard any group that is too short or too sparse to be a real boundary.

For example, using a DashDotBoundaryStyle strategy with 25px tolerance, 17,000+ raw segments collapse into just 15 horizontal + 18 vertical clusters — each representing a candidate panel edge.

Figure 12. After clustering: 15 horizontal lines (left) and 18 vertical lines (right), each color-coded by cluster. Legend shows pixel position and segment count per cluster. The 3,604 unassigned segments (gray) are symbol strokes and other non-boundary elements, filtered out.

# Group collinear, adjacent line segments into chains
# Lines sharing an endpoint (within tolerance) and same direction → one chain
chains = cluster_lines_to_chains(svg_lines, endpoint_tolerance=2.0)

# Filter by length — short chains are symbols, long chains are boundaries
boundary_chains = [c for c in chains if c.total_length > min_boundary_length]

Density-Peak Clustering for Boundary Lines

Dotted lines have gaps too large to bridge by proximity. Instead, project all fragments onto a ruler and count how many land at each position. Wherever fragments pile up — a spike — that's a boundary line. Find the spikes, ignore the noise.

# Find density peaks in X-coordinates of vertical lines
# and Y-coordinates of horizontal lines
x_peaks = density_peak_cluster(vertical_lines, axis='x')
y_peaks = density_peak_cluster(horizontal_lines, axis='y')

# Each peak represents a candidate panel edge position

BoundaryStyleStrategy Abstraction

Different drawings use different boundary conventions — solid lines, dashed lines, mixed styles. A strategy pattern allows the detection algorithm to adapt:

# Pluggable strategy for what constitutes a "boundary line"
class BoundaryStyleStrategy:
    def is_boundary_line(self, line) -> bool: ...
    def merge_candidates(self, lines) -> List[Chain]: ...

# Implementations:
# - SolidLineStrategy: continuous lines above length threshold
# - DashedLineStrategy: periodic short segments with gaps
# - MixedStrategy: combines both heuristics

Seed-Guided Minimal Cycle Extraction

The final step finds the actual closed regions. Using panel name locations as seeds (from Technique 2), we search for the minimal cycle in the planar graph that encloses each seed point:

# Build planar graph from boundary line segments
G = build_planar_graph(boundary_chains)

for panel_name, seed_point in panel_seeds:
    # Find minimal cycle enclosing the seed point
    cycle = find_minimal_enclosing_cycle(G, seed_point)
    if cycle:
        panel_regions[panel_name] = cycle_to_polygon(cycle)

Trade-off: This approach is faster and more precise than LLM-based detection, but requires SVG vector access (not always available) and is sensitive to non-standard drawing conventions. In our pipeline, it serves as an alternative path when vector data is accessible.

Results & Error Analysis

The pipeline achieved 94.21% accuracy (277/294 materials correctly extracted) across 53 panels on 4 diagram pages. Processing time was ~62 minutes from pre-cropped panel images.

Metric	Value
Overall accuracy	94.21%
Panels processed	53
Materials extracted	346
Correctly identified	277 / 294
Processing time	~62 min (panel images only)

A +5.43% accuracy improvement (88.78% → 94.21%) came from iterative prompt refinement based on domain expert review of extraction errors — identifying recurring patterns and translating them into prompt corrections.

Figure 13. Iterative prompt refinement: GPT-5.4 results → expert review → error pattern analysis → prompt corrections → re-run.

Key Takeaways

Decompose aggressively. Stage-wise processing with scoped inputs is the difference between working and not working. Break the problem down until each sub-task is tractable for the model.

Visual context beats reasoning effort. Color-coded overlays and few-shot images reduce reliance on expensive reasoning modes. Optimize inputs before scaling compute.

Verify, but stop early. Self-correction loops improve accuracy — but accumulated context can confuse the model. Oscillation detection and early stopping are critical.

Hybrid always wins. Azure Document Intelligence for precise coordinates, GPT-5,4 for semantics. Pure LLM solutions lose on precision-critical tasks.

What's Next?

The techniques here generalize well beyond SLDs. We're exploring several directions:

Other drawing types — P&ID, mechanical assembly, architectural floor plans. The core pipeline stages stay the same; only the few-shot guides and panel name patterns change per domain.
ERP/PLM integration — Feeding extracted BOMs directly into SAP, Oracle, or PTC Windchill to close the loop from drawing to purchase order.
Active learning from HITL corrections — Using human corrections captured in the Streamlit demo as training signal to drive automatic prompt refinement.
Cost optimization at scale — Batching Azure OpenAI calls, caching DI results for recurring templates, and leveraging SVG vector detection (Technique 6) whenever CAD exports are available.
Multi-modal verification — Cross-referencing extracted BOMs against parts databases or previous drawing revisions to validate extraction accuracy in context.

Get Started

Run the Demo Clone the GitHub repository and launch the Streamlit HITL demo with your own SLD drawings.

Figure 14. The Streamlit HITL demo showing Step 5 — BOM Extraction results. Each panel's cropped image is displayed alongside the extracted component table with device symbol, name, specification, quantity, and confidence level.

View on GitHub →

Explore the Services

Join the Conversation

Have questions or built something similar? Share your experience in the comments below or connect with us on the Azure AI Tech Community.

Related Reading:

Introducing OpenAI's GPT-image-2 in Microsoft Foundry

Naomi Moneypenny — Tue, 21 Apr 2026 20:52:15 GMT

Take a small design team running a global social campaign. They have the creative vision to produce localized imagery for every market, but not the resources to reshoot, reformat, or outsource that scale. Every asset needs to fit a different platform, a different dimension, a different cultural context, and they all need to ship at the same time. This is where flexible image generation comes in handy.

OpenAI's GPT-image-2 is now generally available and rolling out today to Microsoft Foundry, introducing a step change in image generation. Developers and designers now get more control over image output, so a small team can execute with the reach and flexibility of a much larger one.

What is new in GPT-image-2?

GPT-image-2 brings real world intelligence, multilingual understanding, improved instruction following, increased resolution support, and an intelligent routing layer giving developers the tools to scale image generation for production workflows.

Real world intelligence

GPT-image-2 has a knowledge cut off of December 2025, meaning that it is able to give you more contextually relevant and accurate outputs. The model also comes with enhanced thinking capabilities that allow it to search the web, check its own outputs, and create multiple images from just one prompt. These enhancements shift image generation models away from being simple tools and runs them into creative sidekicks.

Multilingual understanding

GPT-image-2 includes increased language support across Japanese, Korean, Chinese, Hindi, and Bengali, as well as new thinking capabilities. This means the model can create images and render text that feels localized.

Increased resolution support

GPT-image-2 introduces 4K resolution support, giving developers the ability to generate rich, detailed, and photorealistic images at custom dimensions.

Resolution guidelines to keep in mind:

Constraint	Detail
Total pixel budget	Maximum pixels in final image cannot exceed 8,294,400 Minimum pixels in final image cannot be less than 655,360 Requests exceeding this are automatically resized to fit.
Resolutions	4K, 1024x1024, 1536x1024, and 1024x1536
Dimension alignment	Each dimension must be a multiple of 16

Note: If your requested resolution exceeds the pixel budget, the service will automatically resize it down.

Intelligent routing layer

GPT-image-2 also includes an expanded routing layer with two distinct modes, allowing the service to intelligently select the right generation configuration for a request without requiring an explicitly set size value.

Mode 1 — Legacy size selection

In Mode 1, the routing layer selects one of the three legacy size tiers to use for generation:

Size tier	Description
smimage	Small image output
image	Standard image output
xlimage	Large image output

This mode is useful for teams already familiar with the legacy size tiers who want to benefit from automatic selection without making any manual changes.

Mode 2 — Token size bucket selection

In Mode 2, the routing layer selects from six token size buckets — 16, 24, 36, 48, 64, 96 — which map roughly to the legacy size tiers:

Token bucket	Approximate legacy size
16, 24	smimage
36, 48	image
64, 96	xlimage

This approach can allow for more flexibility in the number of tokens generated, which in turn helps to better optimize output quality and efficiency for a given prompt.

See it in action

GPT-image-2 shows improved image fidelity across visual styles, generating more detailed and refined images. But, don’t just take our word for it, let's see the model in action with a few prompts and edits. Here is the example we used:

Prompt: Interior of an empty subway car (no people).
Wide-angle view looking down the aisle. Clean, modern subway car with seats, poles, route map strip, and ad frames above the windows.
Realistic lighting with a slight cool fluorescent tone, realistic materials (metal poles, vinyl seats, textured floor).

Figure 1. Created with GPT-image-1

Figure 2. Created with GPT-image-1.5

Figure 3. Created with GPT-image-2

As you can see, when using the same base prompt, the image quality and realism improved with each model. Now let’s take a look at adding incremental changes to the same image:

Prompt: Populate the ad frames with a cohesive ad campaign for “Zava Flower Delivery” and use an array of flower types.

Figure 4. Created with GPT-image-2

And our subway is now full of ads for the new ZAVA flower delivery service. Let's ask for another small change:

Prompt: In all Zava Flower Delivery advertisements, change the flowers shown to roses (red and pink roses).

Figure 5. Created with GPT-image-2

And in three simple prompts, we've created a mockup of a flower delivery ad. From marketing material to website creation to UX design, GPT-image-2 now allows developers to deliver production-grade assets for real business use cases.

Image generation across industries

These new capabilities open the door to richer, more production-ready image generation workflows across a range of enterprise scenarios:

Retail & e-commerce: Generate product imagery at exact platform-required dimensions, from square thumbnails to wide banners, without post-processing.
Marketing: Produce crisp, rich in color campaign visuals and social assets localized to different markets.
Media & entertainment: Generate storyboard panels and scene at resolutions suited to production pipelines.
Education & training: Create visual learning aids and course materials formatted to exact display requirements across devices.
UI/UX design: Accelerate mockup and prototype workflows by generating interface assets at the precise dimensions your design system requires.

Trust and safety

At Microsoft, our mission to empower people and organizations remains constant. As part of this commitment, models made available through Foundry undergo internal reviews and are deployed with safeguards designed to support responsible use at scale. Learn more about responsible AI at Microsoft.

For GPT-image-2, Microsoft applied an in-depth safety approach that addresses disallowed content and misuse while maintaining human oversight. The deployment combines OpenAI’s image generation safety mitigations with Azure AI Content Safety, including filters and classifiers for sensitive content.

Pricing

Model

Offer type

Pricing - Image

Pricing - Text

GPT-image-2

Standard Global

Input Tokens: $8

Cached Input Tokens: $2

Output Tokens: $30

Input Tokens: $5

Cached Input Tokens: $1.25

Output Tokens: $10

Note: All prices are per 1M token.

Getting started

Whether you’re building a personalized retail experience, automating visual content pipelines or accelerating design workflows. GPT-image-2 gives your team the resolution control and intelligent routing to generate images that fit your exact needs. Try the GPT-image-2 in Microsoft Foundry today!

Deploy the model in Microsoft Foundry

Experiment with the model in the Image playground

Read the documentation to learn more

Troubleshooting Microsoft Foundry Accessing On‑Premises APIs Over Private Networking

pankajag — Mon, 20 Apr 2026 13:00:00 GMT

Audience: Azure solution architects, network engineers, and AI practitioners deploying Microsoft Foundry in enterprise, network‑isolated environments.

Scenario: Foundry Agent Service must call an on‑premises (or privately hosted) API over VPN or ExpressRoute using on‑premises corporate DNS. Connectivity works from a virtual machine (VM) in the virtual network (VNet), but fails when the same call is made from a Foundry agent.

This post consolidates common field patterns observed across customer engagements and maps them directly to official Microsoft guidance. It highlights a frequently missed prerequisite for private connectivity: Project and Agent Capability Hosts.

The Repeating Enterprise Pattern

The architecture is familiar:

Microsoft Foundry account and project
Customer‑managed VNet with VPN or ExpressRoute to on‑premises
Corporate (on‑premises) DNS used for API name resolution
A VM in the VNet can successfully resolve and call the on‑prem API
Foundry agents are configured to call the same API (often via an OpenAPI tool)

Despite this, the agent fails with one or more of the following:

DNS resolution failures
Connection timeouts
HTTP 401 or 403 responses
Unexpected backend or proxy URLs appearing in logs

The consistent question is:

Why does this work from a VM in the VNet but not from the Foundry agents?

Key Principle: Foundry Agents Do Not Automatically Run in Your VNet

This is the most important mental model to reset.

Creating a private endpoint for a Foundry Agent Service does not place agent runtime traffic into your VNet. Private endpoints are inbound constructs. Outbound connectivity from an agent only flows through your VNet when specific requirements are met.

The most critical of those requirements is a Capability Host.

What Is a Capability Host?

A Capability Host defines where Foundry capabilities are allowed to execute.

In private networking scenarios, the capability host:

Binds a Foundry project or agent to a customer‑managed subnet
Enables platform‑managed container injection into that subnet
Ensures outbound traffic follows VNet routing, security controls, and DNS configuration

Capability Host Scope

Project Capability Host

Associated at the project level
Applies to all agents in the project
Defines the customer subnet the project is allowed to use

Agent Capability Host

Associated at the individual agent level
Can explicitly bind or override subnet placement
Useful when agents require different isolation boundaries

Key field insight: If no capability host is associated, the agent runtime is not injected into the VNet—even if VPN, ExpressRoute, private endpoints, and on‑prem DNS are correctly configured.

Capability Host Lifecycle (Conceptual)

1 Microsoft Foundry Account ↓ Project ↓ Capability Host ↓ Delegated Agent Subnet (VNet) ↓ Agent Runtime (Container Injection)

Without the capability host step, the chain breaks and the agent executes outside the customer network boundary.

Why On‑Premises DNS Appears Correct—but Still Fails

DNS is where most investigations stall.

Teams typically confirm:

On‑premises DNS resolves the API hostname
VNet DNS settings forward queries to on‑prem DNS
A VM in the subnet resolves and reaches the API

Yet the agent still fails.

The reason is simple:

The VM is unquestionably inside the VNet
The agent may not be

Without a capability host, the agent runtime does not inherit:

VNet DNS server settings
Corporate DNS forwarding rules
On‑premises name resolution paths

As a result, DNS fails even though the DNS design itself is correct.

Secondary Symptom: HTTP 401 Errors

After subnet injection and DNS are corrected, some customers encounter HTTP 401 responses.

This typically means:

The API is now reachable
The request is successfully routed through the private path
Authentication or authorization is failing

At this stage, troubleshooting moves from networking to identity:

Validate the credential or token configured in the Foundry connection
Confirm expected headers, audience, or auth flow
Account for managed proxy hops in the request path

A 401 at this point is progress—it confirms private connectivity is working.

Permissions and Required Services for Capability Hosts

Creating capability hosts requires both the correct Azure role-based access control (RBAC) permissions and the presence of specific dependent services. These prerequisites are frequently overlooked and can silently block capability host creation or leave hosts in a failed provisioning state.

Required Permissions (RBAC)

Microsoft documentation explicitly calls out the following minimum permissions:

Contributor role on the Microsoft Foundry account to create capability hosts.
User Access Administrator or Owner role on the subscription or resource group to grant the Foundry project’s managed identity access to dependent Azure resources when using standard agent setup.

Without these roles, capability host creation may fail, or the host may be created without access to required downstream services. For details, see Role-based access control (RBAC) for Microsoft Foundry.

Required Azure Services and Connections

For standard and network-secured agent setups, capability hosts reference customer-owned Azure resources. The following services must exist and be connected to the Foundry project before creating the capability host:

Azure Storage – for file uploads and artifacts
Azure AI Search – for vector stores and retrieval
Azure Cosmos DB – for thread and conversation storage
Azure AI Services / Azure OpenAI – for model execution

These services must be deployed in supported regions and, for private networking scenarios, in the same region as the virtual network. Capability hosts reference these resources through project connections, not raw resource IDs .

Networking-Specific Requirements

When using private networking:

The agent subnet must already exist and be delegated appropriately.
The capability host must reference the correct customer subnet at creation time.
Required private endpoints and DNS resolution must be in place for dependent services.

If networking or connections change, capability hosts cannot be updated in-place and must be deleted and recreated with the corrected configuration .

Practical Fix Patterns

1. Create and Associate a Project Capability Host

Bind the project to the intended delegated agent subnet
Verify the customerSubnet reference
Redeploy the agent after association

This aligns directly with the Standard Setup with Private Networking model documented by Microsoft.

2. Validate Agent Placement and Network Inheritance

Confirm the agent is associated with the expected capability host
Verify the capability host references the correct subnet
Ensure network and DNS settings are applied at the subnet level

The agent inherits routing and DNS behavior only after successful subnet injection.

3. Validate DNS From the Agent Subnet

Confirm VNet DNS settings point to on‑prem DNS
Test name resolution from a VM in the same subnet

Once injected, the agent uses the same DNS behavior as other resources in that subnet.

4. Use a Supported Foundry Experience

Be aware of documented constraints:

End‑to‑end network isolation is not supported in the newer Foundry portal experience
Network‑isolated agent scenarios require the classic Foundry experience, SDK, or CLI
Hosted agents do not support full isolation

Mismatch here can make a correct network design appear broken.

A Simple Checklist

When a Foundry agent cannot reach an on‑prem API:

Is a **Project Capability **Host associated?
Is the capability host bound to the correct subnet?
Is on‑prem DNS reachable from that subnet?
Is a supported Foundry experience in use?
If reachable, is authentication configured correctly?

If items 1 and 2 are not satisfied, all other troubleshooting is premature.

Closing

Most private networking issues with Microsoft Foundry are not caused by VPNs or DNS infrastructure. They result from an incomplete understanding of **where the agent ****runtime ****actually **executes.

Capability Hosts are the control point. When they are correctly configured, Foundry agents behave exactly as described in Microsoft guidance: they inherit VNet routing, DNS, and security controls and can securely access on‑premises systems over private connectivity.

No capability host = no VNet injection = no on‑prem connectivity.

References

The following Microsoft‑published resources were referenced and aligned with throughout this article:

Network‑secured agent setup (GitHub) – Reference implementation demonstrating a network‑secured Foundry Agent Service deployment with a customer‑managed virtual network, delegated agent subnet, and private connectivity patterns. This notebook illustrates how agent runtimes inherit network behavior only after subnet injection ralacher/network-secured-agent
Set up private networking for Foundry Agent Service - Microsoft Foundry | Microsoft Learns .

Need Guidance on cost breakdown of Microsoft Foundry Agent portal I created

Tasmia_Monzoor — Mon, 20 Apr 2026 07:20:01 GMT

I have developed a complaint handling portal for customers and employees using Azure AI Foundry. The solution is built with Foundry agents, models from the catalog, input/output caching, agent logging/tracing, and other Foundry capabilities. The frontend and orchestration layer are deployed on Azure Container Apps.

While Azure Cost Analysis provides an overview of spending, several parts remain unclear or act as a black box for accurate estimation, including:

Token consumption assumptions (input/output tokens across different models and agents)
User concurrency, sessions, and behavior patterns
Agent logging and observability costs
Impact of input/output caching
Detailed resource consumption and billing in Azure Container Apps

What is the best way to accurately calculate or estimate the total running cost for such an Azure AI Foundry-based platform with Container Apps frontend?

Are there official Microsoft documentation, pricing guides, or reference architectures for cost breakdown? How do companies typically present costs for such AI platforms to attract customers (e.g., TCO models or per-user pricing)? I want to know how the platform costs are shown to customers.

Thank you.

Unable to add SharePoint site as a tool in Foundry Agent (403 – User does not have valid license)

Angela2 — Fri, 17 Apr 2026 13:45:11 GMT

Hi,
I’m very new to Foundry and I’m trying to add a SharePoint site as a tool (SharePoint grounding) in a Foundry Agent, but it fails with:

HTTP 403 – Forbidden Authorization Failed – User does not have valid license Tool: sharepoint_grounding

Error{"error": "Tool_User_Error", "message": "[Sharepoint-tool] Request to Graph API failed with HTTP status 403, error-code: Forbidden and error-message: Authorization Failed - User does not have valid license. Client Request Id: 0000000000000000000000. Find out more troubleshooting details here - https://aka.ms/foundrysharepointtroubleshooting", "code": "sharepoint_grounding_tool_user_error", "tool": "sharepoint_grounding", "allow_retry": false, "extra_info": null}

Azure roles, Graph permissions, and SharePoint access are all correctly configured (Owner, Azure AI Admin/Developer/User), and the SharePoint site is accessible outside Foundry. Despite this, Foundry blocks the tool with a license error.

Any help or guidance would be very much appreciated.
regards
Angela

Claude Opus 4.7 is available on Microsoft Foundry

amar_badal — Thu, 16 Apr 2026 16:20:43 GMT

Anthropic continues to push the frontier of AI for real-world, production work with its new model Claude Opus 4.7. Opus 4.7 is Anthropic’s most advanced generally available model. Teams building on Azure can access Claude Opus 4.7 through Microsoft Foundry , today. This model is designed for the workflows enterprises run and delivers meaningful gains across agentic coding, long-running autonomous tasks, and professional work.

For organizations already building with Claude Opus 4.6, Opus 4.7 represents a powerful upgrade bringing stronger instruction following, better vision and office outputs, and improved memory capabilities to enable enhanced performance across complex, multi-step workflows..

What Microsoft Foundry brings to the table

Microsoft Foundry is Microsoft's unified platform for building, deploying, and governing AI applications at scale. It gives enterprise teams a single control plane for model selection, data connections, observability and access controls all backed by Azure's global infrastructure and security layer.

Bringing Claude Opus 4.7 into that environment means enterprises get frontier model capability without compromising governance properties their security and legal teams require.

Developers call Claude Opus 4.7 through Foundry's standard model APIs, which means existing tool chains, SDKs, and prompt harnesses require minimal changes. For teams already running Opus 4.6 in Microsoft Foundry, this is a direct upgrade path.

What Opus 4.7 adds for enterprise workloads

These improvements land directly in production workflows. Enterprises running Opus 4.6 today should see measurable gains in long-running tasks, coding, visual capabilities, instruction following, and reasoning. .

The model reasons through underspecified requests, making sensible assumptions and stating them clearly rather than stalling on clarifying questions a practical improvement for any workflow that touches real-world data. Check out Anthropic blog for benchmark table

Built-in enterprise governance

Running Opus 4.7 through Microsoft Foundry means every inference request inherits Foundry's enterprise controls: Azure Active Directory for identity and access, private networking via VNet and private endpoints, logging and audit trails through Azure Monitor.

Teams don't need to build a separate governance layer around the model. Foundry handles it, and security teams already know how to audit it.

Key use cases by vertical

Getting started

Claude Opus 4.7 is available today in Microsoft Foundry's model catalog. Teams already using Anthropic models on Microsoft Foundry can upgrade through the catalog with no infrastructure changes. New teams can deploy through the standard Foundry provisioning flow.

Claude Code is available via the Anthropic API routed through Microsoft Foundry; teams can point their existing Claude Code installations at their Foundry endpoint with a single configuration change.

GitHub Copilot CLI with Claude is also available to GitHub Copilot Enterprise subscribers today.

Anthropic has provided a migration guide covering the key differences between Opus 4.6 and Opus 4.7 to help teams get the most out of the new model with minimal harness changes.

Introducing MAI-Image-2-Efficient: Faster, More Efficient Image Generation

Naomi Moneypenny — Tue, 14 Apr 2026 16:00:17 GMT

Building on our momentum

Just last week, we celebrated a major milestone: the public preview launch of three new first-party Microsoft AI models in Microsoft Foundry: MAI-Image-2, MAI-Voice-1, and MAI-Transcribe-1. Together, they represent a comprehensive multimedia AI stack purpose-built for developers that spans image generation, natural speech synthesis, and enterprise-grade transcription across 25 languages.

The response from the developer community has been incredible, and we're not slowing down.

Fast on the heels of that launch, we're thrilled to introduce the next addition to the MAI image generation family: MAI-Image-2-Efficient – or Image-2e for short. It’s now available in public preview in Microsoft Foundry and MAI Playground.

What makes MAI-Image-2-Efficient unique?

MAI-Image-2-Efficient is built on the same architecture as MAI-Image-2 which is the model that debuted at #3 on the Arena.ai leaderboard for image model families. Based on customer feedback, we’ve now improved it and engineered for speed and efficiency.

It’s up to 22% faster with 4x more efficiency compared to MAI-Image-2 when normalized by latency and GPU usage¹. It also outpaces leading text-to-image models by 40% on average².

In short, MAI-Image-2-Efficient gives developers more output for less compute, unlocking a whole new category of use cases.

Who is MAI-Image-2-Efficient for?

MAI-Image-2-Efficient is designed for builders who need high-quality image generation at speed and scale. Here are the top use cases where Image-2-Efficient shines:

High-volume production workflows: E-commerce platforms, media companies, and marketing teams often need to generate thousands of images per day, as part of their business processes to generate targeted advertisements, concept art and mood boards. MAI-Image-2-Efficient's superior efficiency means larger batches at lower GPU cost, so your team can think and iterate as fast as you want and reach the end-product faster.
Real-time and conversational experiences: When users expect images to appear mid-conversation (in a chatbot, a creative copilot, or an AI-powered design tool), every millisecond counts. Thanks to its lower latency, MAI-Image-2-Efficient serves as an excellent backbone for interactive applications that require fast response times.
Rapid prototyping and creative iteration: MAI-Image-2-Efficient enables your team to quickly and affordably test new pipelines, experiment with creative ideas, or refine prompts. You don't need the complete model to validate a concept; what you need is speed, and that's exactly what MAI-Image-2-Efficient provides.

MAI-Image-2 vs. MAI-Image-2-Efficient — which should you use?

MAI-Image-2-Efficient and MAI-Image-2 are built for different strengths, so choosing the right model depends on the needs of your workflow.

MAI-Image-2-Efficient is the ideal choice for high-volume workflows where latency and speed are priorities. If your pipeline needs to generate images quickly and at scale, MAI-Image-2-Efficient delivers without compromise.

MAI-Image-2 is the recommended option when your images require precise, detailed text rendering, or when scenes demand the deepest photorealistic contrast and smoothness.

The two models also have distinct visual signatures:

MAI-Image-2-Efficient renders with sharpness and defined lines, making it a strong choice for illustration, animation, and photoreal images designed to grab attention.

MAI-Image-2 delivers smoother, more nuanced contrast, making it the go-to for photorealistic imagery that prioritizes depth and subtlety.

Try it today

MAI-Image-2-Efficient is available now in Microsoft Foundry and MAI Playground. For builders in Foundry, MAI-Image-2-Efficient starts at $5 USD per 1M tokens for text input and $19.50 USD per 1M tokens for image output.

And this is just the beginning. We have more exciting announcements lined up; stay tuned for what we're bringing to Microsoft Build 2026.

References:

As tested on April 13, 2026. Compared to MAI-Image-2 when normalized by latency and GPU usage. Throughput per GPU vs MAI-Image-2 on NVIDIA H100 at 1024×1024; measured with optimized batch sizes and matched latency targets. Results vary with batch size, concurrency, and latency constraints.
As tested on April 13, 2026. Compared to Gemini 3.1 Flash (high reasoning), Gemini 3.1 Flash Image and Gemini 3 Pro Image: Measured at p50 latency via AI Studio API (1:1, 1K images; minimal reasoning unless noted; web search disabled). MAI-Image-2, MAI-Image-2-Efficient, GPT-Image-1.5-High: Measured at p50 latency via Foundry API.

Gemma 4 now available in Microsoft Foundry

vaidyas — Tue, 14 Apr 2026 16:05:26 GMT

Experimenting with open-source models has become a core part of how innovative AI teams stay competitive: experimenting with the latest architectures and often fine-tuning on proprietary data to achieve lower latencies and cost.

Today, we’re happy to announce that the Gemma 4 family, Google DeepMind’s newest model family, is now available in Microsoft Foundry via the Hugging Face collection.

Azure customers can now discover, evaluate, and deploy Gemma 4 inside their Azure environment with the same policies they rely on for every other workload. Foundry is the only hyperscaler platform where developers can access OpenAI, Anthropic, Gemma, and over 11,000+ models under a single control plane. Through our close collaboration with Hugging Face, Gemma 4 joining that collection continues Microsoft’s push to bring customers the widest selection of models from any cloud – and fits in line with our enhanced investments in open-source development.

Frontier Intelligence, open-source weights

Released by Google DeepMind on April 2, 2026, Gemma 4 is built from the same research foundation as Gemini 3 and packaged as open weights under an Apache 2.0 license.

Key capabilities across the Gemma 4 family:

Native multimodal: Text + image + video inputs across all sizes; analyze video by processing sequences of frames; audio input on edge models (E2B, E4B)
Enhanced reasoning & coding capabilities: Multi-step planning, deep logic, and improvements in math and instruction-following enabling autonomous agents
Trained for global deployment: Pretrained on 140+ languages with support for 35+ languages out of the box
Long context: Context windows of up to 128K tokens (E2B/E4B) and 256K tokens (26B A4B/31B) allow developers to reason across extensive codebases, lengthy documents, or multi-session histories

Why choose Foundry?

Foundry is built to give developers breadth -- access to models from major model providers, open and proprietary, under one roof.

Stay within Azure to work leading models. When you deploy through Foundry, models run inside your Azure environment and are subject to the same network policies, identity controls, and audit processes your organization already has in place.
Managed online endpoints handle serving, scaling, and monitoring without manually setting up and managing the underlying infrastructure.
Serverless deployment with Azure Container Apps allows developers to deploy and run containerized applications while reducing infrastructure management and saving costs.
Gated model access integrates directly with Hugging Face user tokens, so models that require license acceptance stay compliant can be accessed without manual approvals.
Foundry Local lets you run optimized Hugging Face models directly on your own hardware using the same model catalog and SDK patterns as your cloud deployments. Read the documentation here: https://aka.ms/foundrylocal and https://aka.ms/HF/foundrylocal
Microsoft’s approach to Responsible AI is grounded in our AI principles of fairness, reliability and safety, privacy and security, inclusiveness, transparency, and accountability. Microsoft Foundry provides governance controls, monitoring, and evaluation capabilities to help organizations deploy new models responsibly in production environments.

What are teams building with Gemma 4 in Foundry

Gemma 4’s combination of multimodal input, agentic function calling, and long context offers a wide range of production use cases:

Document intelligence: Processing PDFs, charts, invoices, and complex tables using native vision capabilities
Multilingual enterprise apps: 140+ natively trained languages — ideal for multinational customer support, content platforms as well as language learning tools for grammar correction and writing practice
Long-context analytics: Reasoning across entire codebases, legal documents, or multi-session conversation histories

Getting started

Try Gemma 4 in Microsoft Foundry today. New models from Hugging Face continue to roll out to Foundry on a regular basis through our ongoing collaboration. If there's a model you want to see added, let us know here. Stay connected to our developer community on Discord and stay up to date on what is new in Foundry through the Model Mondays series.

Building Real-Time Speech Translation with AI Avatars with Azure Speech Services

sudarsan — Tue, 14 Apr 2026 15:45:00 GMT

Introduction

Language barriers remain one of the biggest challenges in communication. Whether you are holding an all-hands meeting for a globally distributed team, consulting with non-native speaking patients, or teaching students across continents—seamless, real-time translation makes or breaks effective communication.Traditional translation tools feel impersonal and disconnected. Text captions scroll across screens while speakers continue in their native tongue, creating a disjointed experience. What if your audience could see and hear an AI avatar speaking directly to them in their own language, with natural lip-sync and human-like expressions?

We can use Azure Speech Translation and Avatar to address this: a speaker talks in one language, and listeners watch an AI avatar deliver the translated speech in their chosen language. Imagine a CEO in Tokyo delivering a quarterly update. Employees in Munich, São Paulo, and Mumbai each see an AI avatar speaking to them in German, Portuguese, and Hindi respectively—all in real-time, with synchronized lip movements and natural speech patterns. The speaker focuses on their message; the technology handles the rest.

In this blog, we will discuss a sample implementation that used Azure Speech, Translation and Avatar capabilities.

How It Works

📚 Ready to build your own real-time translation avatar application? Grab the complete source code and documentation from GitHub : github.com/l-sudarsan/avatar-translation

The application uses a session-based Speaker/Listener architecture to separate the presenter's control interface from the audience's viewing experience. The speaker can create and configure a session based on requirements

Speaker Mode

The speaker interface gives presenters full control over the translation session:

Session Management: Create sessions and generate shareable listener URLs
Language Configuration: Select source language (what you speak) and target language (what listeners hear)
Avatar Selection: Choose from prebuilt or custom avatars for the translation output
Real-time Feedback: View live transcription of your speech and monitor listener count
No Avatar Display: The interface intentionally hides the avatar video/audio to prevent microphone feedback loops

Listener Mode

The listener interface delivers an immersive, distraction-free viewing experience:

Easy Access: Join via a simple URL containing the session code (e.g., /listener/123456)
Avatar Video: Watch the AI avatar with synchronized lip movements matching the translated speech
Translated Audio: Hear the avatar speak the translation in the target language
Caption Display: Read real-time translation text alongside the avatar
Translation History: Scroll through all translations from the session

Data Flow & Solution Components

The diagram below shows data flow and how the components interact. The Flask server acts as the central hub, coordinating communication between the speaker's browser, Azure Speech Services, and multiple listener clients.

Implementation Deep Dive

You can check the complete source code in the GitHub repository.

Core Components

Five main technical components power the application, each handling a specific part of the translation pipeline.

1. Backend: Flask + Socket.IO

The server uses Flask and Flask-SocketIO with the Eventlet async worker for WebSocket support. This combination delivers:

HTTP endpoints for session management and avatar connection
WebSocket rooms for real-time translation broadcasting
Session storage for managing multiple concurrent translation sessions

# Session structure
sessions = {
    "123456": {
        "name": "Q1 Townhall",
        "source_language": "en-US",
        "target_language": "ja-JP",
        "avatar": "lisa",
        "listeners": set(),
        "is_translating": False
    }
}

2. Audio Streaming: Browser to Server

Instead of relying on server-side microphone access, the browser captures audio directly using the Web Audio API:

// Speaker captures microphone at 16kHz
const audioContext = new AudioContext({ sampleRate: 16000 });
const mediaStream = await navigator.mediaDevices.getUserMedia({ audio: true });

// Process audio and send via Socket.IO
processor.onaudioprocess = (event) => {
    const pcmData = convertToPCM16(event.inputBuffer);
    socket.emit('audioData', { sessionId, audioData: pcmData });
};

This approach works seamlessly across different deployment environments without requiring server microphone permissions.

3. Azure Speech Translation

The server receives audio chunks and feeds them to Azure's TranslationRecognizer via a PushAudioInputStream:

# Configure translation
translation_config = speechsdk.translation.SpeechTranslationConfig(
    subscription=SPEECH_KEY,
    region=SPEECH_REGION
)
translation_config.speech_recognition_language = "en-US"
translation_config.add_target_language("ja")

# Push audio stream
push_stream = speechsdk.audio.PushAudioInputStream()
audio_config = speechsdk.audio.AudioConfig(stream=push_stream)

# Handle recognition results
def on_recognized(evt):
    translation = evt.result.translations["ja"]
    socketio.emit('translationResult', {
        'original': evt.result.text,
        'translated': translation
    }, room=session_id)

4. Avatar Synthesis with WebRTC

Each listener establishes a WebRTC connection to Azure's Avatar Service:

ICE Token Exchange: Server provides TURN server credentials
SDP Negotiation: Browser and Azure exchange session descriptions
Avatar Connection: Listener sends local SDP offer, receives remote answer
Video Stream: Avatar video flows directly to listener via WebRTC

// Listener connects to avatar
const peerConnection = new RTCPeerConnection(iceConfig);
const offer = await peerConnection.createOffer();
await peerConnection.setLocalDescription(offer);

// Send to Azure Avatar Service
const response = await fetch('/api/connectListenerAvatar', {
    method: 'POST',
    headers: { 'session-id': sessionId },
    body: JSON.stringify({ sdp: offer.sdp })
});

const { sdp: remoteSdp } = await response.json();
await peerConnection.setRemoteDescription({ type: 'answer', sdp: remoteSdp });

5. Real-Time Broadcasting

When the speaker talks, translations flow to all listeners simultaneously:

Each listener maintains their own WebRTC connection to the Avatar Service, ensuring independent video streams while receiving synchronized translation text.

WebRTC Avatar Connection Flow

The avatar video streaming uses WebRTC for low-latency delivery. Each listener establishes their own peer connection to Azure's Avatar Service through a multi-step handshake process.

Key Design Decisions

Browser audio capture: Works in any environment without requiring server microphone permissions
Session-based rooms: Isolates translation streams and supports multiple concurrent sessions
Separate speaker/listener UIs: Prevents audio feedback and optimizes each user's experience
Socket.IO for broadcasts: Delivers reliable real-time messaging with automatic reconnection
WebRTC for avatar: Provides low-latency video streaming with peer-to-peer efficiency

Application Areas

Real-time speech translation with AI avatars unlocks transformative possibilities across industries. Here are key sectors where this technology drives significant impact.

🏢 Enterprise & Corporate

Internal Townhalls & All-Hands Meetings

Global organizations deliver executive communications where every employee hears the message in their native language—not through subtitles, but through an avatar speaking directly to them.

Sales Conversations

Sales teams engage international prospects without language barriers. The avatar builds a more personal connection than text translation while preserving the original speaker's authenticity.

Training & Onboarding

Standardized training content reaches employees worldwide, with each viewer experiencing the material in their preferred language through an engaging avatar presenter.

🏥 Healthcare

Patient Communication

Healthcare providers consult with patients who speak different languages, while the avatar delivers medical information clearly and accurately in the patient's native tongue.

Telehealth

Remote healthcare consultations reach non-native speakers effectively, improving health outcomes by ensuring patients fully understand their care instructions.

🎓 Education

Online Learning

Educational institutions expand their global reach, offering lectures and courses in multiple languages through avatar presenters.

Interactive Lessons

Engaging avatar presenters captivate students while delivering content in their native language.

Museum Tours

Cultural institutions offer multilingual guided experiences where visitors receive personalized tours in their language of choice.

📺 Media & Entertainment

Broadcasting

News organizations and content creators deliver content to international audiences with localized avatar presenters, keeping viewers engaged while breaking language barriers.

Live Events

Conferences, product launches, and presentations reach global audiences with real-time translated avatar streams for each language group.

Custom Avatars: Your Brand, Your Voice

While prebuilt avatars work great for many scenarios, organizations can build custom avatars that represent their brand identity. This section covers the creation process and important ethical considerations.

The Process

Request Access: Submit Microsoft's intake form for custom avatar approval
Record Training Data: Capture at least 10 minutes of video featuring your avatar talent
Obtain Consent: Record the talent acknowledging use of their likeness
Train the Model: Use Microsoft Foundry Portal to train your custom avatar
Deploy: Deploy the trained model to your Azure Speech resource

Responsible AI Considerations

Building synthetic representations of people carries ethical responsibilities:

Explicit Written Consent: Always get permission from the talent
Informed Consent: Make sure talent understands how the technology works
Usage Transparency: Share intended use cases with the talent
Prohibited Uses: Never use for deception, misinformation, or impersonation

Microsoft publishes comprehensive Responsible AI guidelines that you must follow when creating custom avatars.

Getting Started

Ready to build your own real-time translation avatar application? Grab the complete source code and documentation from GitHub.

📚 Full Documentation: github.com/l-sudarsan/avatar-translation/docs

Prerequisites

Python 3.8+
Azure Speech Service subscription
Modern browser (Chrome, Edge, Firefox)

Quick Start

# Clone the repository
git clone https://github.com/l-sudarsan/avatar-translation.git
cd avatar-translation

# 1. Create and activate virtual environment
python -m venv venv
.\venv\Scripts\Activate

# 2. Install dependencies
pip install -r requirements.txt

# 3. Configure Azure credentials
cp .env.example .env
# Edit .env with your SPEECH_REGION and SPEECH_KEY

# 4. Run the application
python -m flask run --host=0.0.0.0 --port=5000

Demo Sequence

Open http://localhost:5000/speaker
Configure session (name, source language, target language, avatar)
Click Create Session → Copy the listener URL
Open the listener URL in another browser/device
Wait for the avatar to connect (video appears)
Start speaking → Listeners see the avatar + translations

Tip: For the best demo experience, open the listener URL on a separate device to avoid audio feedback from the avatar's output being picked up by the speaker's microphone.

Conclusion

Real-time speech translation with AI avatars marks a significant leap forward in cross-language communication. By combining Azure's powerful Speech Translation, Text-to-Speech, and Avatar Synthesis services, you can build experiences that feel personal and engaging—not just functional.

The future of multilingual communication isn't about reading subtitles. It's about having someone speak directly to you in your language.

Resources

Project Repository

💻 GitHub: avatar-translation
📚 Project Documentation

Azure Documentation

Simplifying Image Classification with Azure AutoML for Images: A Practical Guide

Serge_Retkowsky — Tue, 14 Apr 2026 14:03:39 GMT

1. The Challenge of Traditional Image Classification

Anyone who has worked with computer vision knows the drill: you need to classify images, so you dive into TensorFlow or PyTorch, spend days architecting a convolutional neural network, experiment with dozens of hyperparameters, and hope your model generalizes well. It’s time-consuming, requires deep expertise, and often feels like searching for a needle in a haystack.

What if there was a better way?

2. Enter Azure AutoML for Images

Azure AutoML for Images is a game-changer in the computer vision space. It’s a feature within Azure Machine Learning that automatically builds high-quality vision models from your image data with minimal code. Think of it as having an experienced ML engineer working alongside you, handling all the heavy lifting while you focus on your business problem.

What Makes AutoML for Images Special?

1. Automatic Model Selection

Instead of manually choosing between ResNet, EfficientNet, or dozens of other architectures, AutoML for Images (Azure ML) evaluates multiple state-of-the-art deep learning models and selects the best one for your specific dataset. It’s like having access to an entire model zoo with an intelligent curator.

2. Intelligent Hyperparameter Tuning

The system doesn’t just pick a model — it optimizes it. Learning rates, batch sizes, augmentation strategies, and more are automatically tuned to squeeze out the best possible performance. What would take weeks of manual experimentation happens in hours.

3. Built-in Best Practices

Data preprocessing, augmentation techniques, and training strategies that would require extensive domain knowledge are pre-configured and applied automatically. You get enterprise-grade ML without needing to be an ML expert.

Key Capabilities

The repository demonstrates several powerful features:

Multi-class and Multi-label Classification: Whether you need to classify an image into a single category or tag it with multiple labels, AutoML manages both scenarios seamlessly.
Format Flexibility: Works with standard image formats including JPEG and PNG, making it easy to integrate with existing datasets.
Full Transparency: Unlike black-box solutions, you maintain complete visibility and control over the training process. You can monitor metrics, understand model decisions, and fine-tune as needed.
Production-Ready Deployment: Once trained, models can be easily deployed to Azure endpoints, ready to serve predictions at scale.

Real-World Applications

The practical applications are vast:

E-commerce: Automatically categorize product images for better search and recommendations.
Healthcare: Classify medical images for diagnostic support.
Manufacturing: Detect defects in production line images.
Agriculture: Identify crop diseases or estimate yield from aerial imagery.
Content Moderation: Automatically flag inappropriate visual content.

3. A Practical Example: Metal Defect Detection

The repository includes a complete end-to-end example of detecting defects in metal surfaces — a critical quality control task in manufacturing. The notebooks demonstrate how to:

Download and organize image data from sources like Kaggle,
Create training and validation splits with proper directory structure,
Upload data to Azure ML as versioned datasets,
Configure GPU compute that scales based on demand,
Train multiple models with automated hyperparameter tuning,
Evaluate results with comprehensive metrics and visualizations,
Deploy the best model as a production-ready REST API,
Export to ONNX for edge deployment scenarios.

The metal defect use case is particularly instructive because it mirrors real industrial applications where quality control is critical but expertise is scarce. The notebooks show how a small team can build production-grade computer vision systems without a dedicated ML research team.

Getting Started: What You Need

The prerequisites are straightforward:

An Azure subscription (free tier available for experimentation)
An Azure Machine Learning workspace
Python 3.7 or later

That’s it. No local GPU clusters to configure, no complex deep learning frameworks to master.

Repository Structure

The repository is thoughtfully organized into three progressive notebooks:

Downloading images.ipynb

Shows how to acquire and prepare image datasets
Demonstrates proper directory structure for classification tasks
Includes data exploration and visualization techniques

image-classification-azure-automl-for-images/1. Downloading images.ipynb at main · retkowsky/image-classification-azure-automl-for-images

Azure ML AutoML for Images.ipynb

The core workflow: connect to Azure ML, upload data, configure training
Covers both simple model training and advanced hyperparameter tuning
Shows how to evaluate models and select the best performing one
Demonstrates deployment to managed online endpoints

image-classification-azure-automl-for-images/2. Azure ML AutoML for Images.ipynb at main · retkowsky/image-classification-azure-automl-for-images

Edge with ONNX local model.ipynb

Exports trained models to ONNX format
Shows how to run inference locally without cloud connectivity
Perfect for edge computing and IoT scenarios

image-classification-azure-automl-for-images/3. Edge with ONNX local model.ipynb at main · retkowsky/image-classification-azure-automl-for-images

Each Python notebook is self-contained with clear explanations, making it easy to understand each step of the process. You can run them sequentially to build a complete solution, or jump to specific sections relevant to your use case.

The Developer Experience

What sets this approach apart is the developer experience. The repository provides Python notebooks that guide you through the entire workflow. You’re not just reading documentation — you’re working with practical, runnable examples that demonstrate real scenarios.

Let’s walk through the code to see how straightforward this actually is.

Use-case description

This image classification model is designed to identify and classify defects on metal surfaces in a manufacturing context.

We want to classify defective images into Crazing, Inclusion, Patches, Pitted, Rolled & Scratches.

Press enter or click to view image in full size

All code and images are available here: retkowsky/image-classification-azure-automl-for-images: Azure AutoML for images — Image classification

Step 1: Connect to Azure ML Workspace

First, establish connection to your Azure ML workspace using Azure credentials:

print("Connection to the Azure ML workspace…") credential = DefaultAzureCredential() ml_client = MLClient( credential, os.getenv("subscription_id"), os.getenv("resource_group"), os.getenv("workspace") ) print("✅ Done")

That’s it.

Step 2: Upload Your Dataset

Upload your image dataset to Azure ML. The code handles this elegantly:

my_images = Data( path=TRAIN_DIR, type=AssetTypes.URI_FOLDER, description="Metal defects images for images classification", name="metaldefectimagesds", ) uri_folder_data_asset = ml_client.data.create_or_update(my_images) print("🖼️ Informations:") print(uri_folder_data_asset) print("\n🖼️ Path to folder in Blob Storage:") print(uri_folder_data_asset.path)

Your local images are now versioned data assets in Azure, ready for training.

Step 3: Create GPU Compute Cluster

AutoML needs compute power. Here’s how you create a GPU cluster that auto-scales:

compute_name = "gpucluster" try: _ = ml_client.compute.get(compute_name) print("✅ Found existing Azure ML compute target.") except ResourceNotFoundError: print(f"🛠️ Creating a new Azure ML compute cluster '{compute_name}'…") compute_config = AmlCompute( name=compute_name, type="amlcompute", size="Standard_NC16as_T4_v3", # GPU VM idle_time_before_scale_down=1200, min_instances=0, # Scale to zero when idle max_instances=4, ) ml_client.begin_create_or_update(compute_config).result() print("✅ Done")

The cluster scales from 0 to 4 instances based on workload, so you only pay for what you use.

Step 4: Configure AutoML Training

Now comes the magic. Here’s the entire configuration for an AutoML image classification job using a specific model (here a resnet34). It is possible as well to access all the available models from the image classification AutoML library.

Press enter or click to view image in full size

https://learn.microsoft.com/en-us/azure/machine-learning/how-to-auto-train-image-models?view=azureml-api-2&tabs=python#supported-model-architectures

image_classification_job = automl.image_classification( compute=compute_name, experiment_name=exp_name, training_data=my_training_data_input, validation_data=my_validation_data_input, target_column_name="label", ) # Set training parameters image_classification_job.set_limits(timeout_minutes=60) image_classification_job.set_training_parameters(model_name="resnet34")

That’s approximately 10 lines of code to configure what would traditionally require hundreds of lines and deep expertise.

Step 5: Hyperparameter Tuning (Optional)

Want to explore multiple models and configurations?

image_classification_job = automl.image_classification( compute=compute_name, # Compute cluster experiment_name=exp_name, # Azure ML job training_data=my_training_data_input, # Training validation_data=my_validation_data_input, # Validation target_column_name="label", # Target primary_metric=ClassificationPrimaryMetrics.ACCURACY, # Metric tags={"usecase": "metal defect", "type" : "computer vision", "product" : "azure ML", "ai": "image classification", "hyper": "YES"}, ) image_classification_job.set_limits( timeout_minutes=60, # Timeout in min max_trials=5, # Max model number max_concurrent_trials=2, # Concurrent training ) image_classification_job.extend_search_space([ SearchSpace( model_name=Choice(["vitb16r224", "vits16r224"]), learning_rate=Uniform(0.001, 0.01), # LR number_of_epochs=Choice([15, 30]), # Epoch ), SearchSpace( model_name=Choice(["resnet50"]), learning_rate=Uniform(0.001, 0.01), # LR layers_to_freeze=Choice([0, 2]), # Layers to freeze ), ]) image_classification_job.set_sweep( sampling_algorithm="Random", # Random sampling to select combinations of hyperparameters. early_termination=BanditPolicy(evaluation_interval=2, # The model is evaluated every 2 iterations. slack_factor=0.2, # If a run’s performance is 20% worse than the best run so far, it may be terminated. delay_evaluation=6), # The policy waits until 6 iterations have completed before starting to # evaluate and potentially terminate runs. )

AutoML will now automatically try different model architectures, learning rates, and augmentation strategies to find the best configuration.

Step 6: Launch Training

Submit the job and monitor progress:

# Submit the job returned_job = ml_client.jobs.create_or_update(image_classification_job) print(f"✅ Created job: {returned_job}") # Stream the logs in real-time ml_client.jobs.stream(returned_job.name)

While training runs, you can monitor metrics, view logs, and track progress through the Azure ML Studio UI or programmatically.

Step 7: Results

Step 8: Deploy to Production

Once training completes, deploy the best model as a REST endpoint:

# Create endpoint configuration online_endpoint_name = "metal-defects-classification" endpoint = ManagedOnlineEndpoint( name=online_endpoint_name, description="Metal defects image classification", auth_mode="key", tags={ "usecase": "metal defect", "type": "computer vision" }, ) # Deploy the endpoint ml_client.online_endpoints.begin_create_or_update(endpoint).result()

Your model is now a production API endpoint, ready to classify images at scale.

Beyond the Cloud: Edge Deployment with ONNX

One of the most powerful aspects of this approach is flexibility in deployment. The repository includes a third notebook demonstrating how to export your trained model to ONNX (Open Neural Network Exchange) format for edge deployment.

This means you can:

Deploy models on IoT devices for real-time inference without cloud connectivity
Reduce latency by processing images locally on edge hardware
Lower costs by eliminating constant cloud API calls
Ensure privacy by keeping sensitive images on-premises

The ONNX export process is straightforward and integrates seamlessly with the AutoML workflow. Your cloud-trained model can run anywhere ONNX Runtime is supported — from Raspberry Pi devices to industrial controllers.

import onnxruntime # Load the ONNX model session = onnxruntime.InferenceSession("model.onnx") # Run inference locally results = session.run(None, {input_name: image_data})

This cloud-to-edge workflow is particularly valuable for manufacturing, retail, and remote monitoring scenarios where edge processing is essential.

Interactive webapp for image classification

Interpreting model predictions

Deployed endpoint returns base64 encoded image string if both model_explainability and visualizations are set to True.

Why This Matters?

In the AI era, the competitive advantage isn’t about who can build the most complex models — it’s about who can deploy effective solutions fastest. Azure AutoML for Images democratizes computer vision by making sophisticated ML accessible to a broader audience.

Small teams can now accomplish what previously required dedicated ML specialists. Prototypes that took months can be built in days. And the quality? Often on par with or better than manually crafted solutions, thanks to AutoML’s systematic approach and access to cutting-edge techniques.

What the Code Reveals

Looking at the actual implementation reveals several important insights:

Minimal Boilerplate: The entire training pipeline — from data upload to model deployment — requires less than 50 lines of meaningful code. Compare this to traditional PyTorch or TensorFlow implementations that often exceed several hundred lines.
Built-in Best Practices: Notice how the code automatically manages concerns like data versioning, experiment tracking, and compute auto-scaling. These aren’t afterthoughts — they’re integral to the platform.
Production-Ready from Day One: The deployed endpoint isn’t a prototype. It includes authentication, scaling, monitoring, and all the infrastructure needed for production workloads. You’re building production systems, not demos.
Flexibility Without Complexity: The simple API hides complexity without sacrificing control. Need to specify a particular model architecture? One parameter. Want hyperparameter tuning? Add a few lines. The abstraction level is perfectly calibrated.
Observable and Debuggable: The `.stream()` method and comprehensive logging mean you’re never in the dark about what’s happening. You can monitor training progress, inspect metrics, and debug issues — all critical for real projects.

The Cost of Complexity

Traditional ML projects fail not because of technology limitations but because of complexity. The learning curve is steep, the iteration cycles are long, and the resource requirements are high. By abstracting away this complexity, AutoML for Images changes the economics of computer vision projects.

You can now:

Validate ideas quickly: Test whether image classification solves your problem before committing significant resources
Iterate faster: Experiment with different approaches in hours rather than weeks
Scale expertise: Enable more team members to work with computer vision, not just ML specialists

Conclusion

Image classification is a fundamental building block for countless AI applications. Azure AutoML for Images makes it accessible, practical, and production-ready. Whether you’re a seasoned data scientist looking to accelerate your workflow or a developer taking your first steps into computer vision, this approach offers a compelling path forward.

The future of ML isn’t about writing more complex code — it’s about writing smarter code that leverages powerful platforms to deliver business value faster. This repository shows you exactly how to do that.

Practical Tips from the Code

After reviewing the notebooks, here are some key takeaways for your own projects:

Start with a Single Model: The basic configuration with `model_name=”resnet34"` is perfect for initial experiments. Only move to hyperparameter sweeps once you’ve validated your data and use case.
Use Tags Strategically: The code demonstrates adding tags to jobs and endpoints (e.g., `”usecase”: “metal defect”`). This becomes invaluable when managing multiple experiments and models in production.
Leverage Auto-Scaling: The compute configuration with `min_instances=0` means you’re not paying for idle resources. The cluster scales up when needed and scales down to zero when idle.
Monitor Training Live: The `ml_client.jobs.stream()` method is your best friend during development. You see exactly what’s happening and can catch issues early.
Version Your Data: Creating named data assets (`name=”metaldefectimagesds”`) means your experiments are reproducible. You can always trace back which data version produced which model.
Think Cloud-to-Edge: Even if you’re deploying to the cloud initially, the ONNX export capability gives you flexibility for future edge scenarios without retraining.

Resources

Azure ML: https://azure.microsoft.com/en-us/products/machine-learning
Demos notebooks: https://github.com/retkowsky/image-classification-azure-automl-for-images
AutoML for Images documentation: https://learn.microsoft.com/en-us/azure/machine-learning/how-to-auto-train-image-models
Available models: Set up AutoML for computer vision — Azure Machine Learning | Microsoft Learn
Connect with the author: https://www.linkedin.com/in/serger/

Microsoft Foundry Agent via Responses API rejects local image input as Base64 data URL / byte array

twennemann — Tue, 14 Apr 2026 07:59:35 GMT

Hello everyone,

we are seeing an issue with the new Microsoft Foundry Agents via the Responses API when sending a local image as part of the user message.

What works

text-only input
image by public URL

What fails

local PNG passed as Base64 data URL
local PNG passed as raw byte array through SDK methods

Example failing image part:

{ "type": "input_image", "image_url": "data:image/png;base64,...", "detail": "auto" }

Returned error:

{ "code": "invalid_payload", "message": "The provided data does not match the expected schema", "param": "/", "type": "invalid_request_error", "details": [] }

We reproduced this in:

C#
Python
raw REST

So this does not appear to be limited to one SDK.

Also important: the same pattern is used in the sample repo for the Foundry Agent Web App, and this scenario worked for us about one week ago:
https://github.com/microsoft-foundry/foundry-agent-webapp

Could you confirm whether local image input is currently supported for Foundry Agents through the Responses API, or whether this is a regression?

Best regards

Now in Foundry: Microsoft Harrier and NVIDIA EGM-8B

Osi — Mon, 13 Apr 2026 20:01:16 GMT

This week's Model Mondays edition highlights three models that share a common thread: each achieves results comparable to larger leading models, as a result of targeted training strategies rather than scale. Microsoft Research's harrier-oss-v1-0.6b from achieves state-of-the-art results on the Multilingual MTEB v2 embedding benchmark at 0.6B parameters through contrastive learning and knowledge distillation. NVIDIA's EGM-8B scores 91.4 average IoU on the RefCOCO visual grounding benchmark by training a small Vision Language Model (VLM) with reinforcement learning to match the output quality of much larger models.

Together they represent a practical argument for efficiency-first model development: the gap between small and large models continues to narrow when training methodology is the focus rather than parameter count alone.

Models of the week

Microsoft Research: harrier-oss-v1-0.6b

Model Specs

Parameters / size: 0.6B
Context length: 32,768 tokens
Primary task: Text embeddings (retrieval, semantic similarity, classification, clustering, reranking)

Why it's interesting

State-of-the-art on Multilingual MTEB v2 from Microsoft Research: harrier-oss-v1-0.6b is a new embedding model released by Microsoft Research, achieving a 69.0 score on the Multilingual MTEB v2 (Massive Text Embedding Benchmark) leaderboard—placing it at the top of its size class at release. It is part of the harrier-oss family spanning harrier-oss-v1-270m (66.5 MTEB v2), harrier-oss-v1-0.6b (69.0), and harrier-oss-v1-27b (74.3), with the 0.6B variant further trained with knowledge distillation from the larger family members. Benchmarks: Multilingual MTEB v2 Leaderboard.
Decoder-only architecture with task-instruction queries: Unlike most embedding models that use encoder-only transformers, harrier-oss-v1-0.6b uses a decoder-only architecture with last-token pooling and L2 normalization. Queries are prefixed with a one-sentence task instruction (e.g., "Instruct: Retrieve relevant passages that answer the query\nQuery: ...") while documents are encoded without instructions—allowing the same deployed model to be specialized for retrieval, classification, or similarity tasks through the prompt alone.
Broad task coverage across six embedding scenarios: The model is trained and evaluated on retrieval, clustering, semantic similarity, classification, bitext mining, and reranking—making it suitable as a general embedding backbone for multi-task pipelines rather than a single-use retrieval model. One endpoint, consistent embeddings across the stack.
100+ language support: Trained on a large-scale mixture of multilingual data covering Arabic, Chinese, Japanese, Korean, and 100+ additional languages, with strong cross-lingual transfer for tasks that span language boundaries.

Try it

Use Case	Prompt Pattern
Multilingual semantic search	Prepend task instruction to query; encode documents without instruction; rank by cosine similarity
Cross-lingual document clustering	Embed documents across languages; apply clustering to group semantically related content
Text classification with embeddings	Encode labeled examples + new text; classify by nearest-neighbor similarity in embedding space
Bitext mining	Encode parallel corpora in source and target languages; align segments by embedding similarity

Sample prompt for a global enterprise knowledge base deployment:

You are building a multilingual internal knowledge base for a global professional services firm. Using the harrier-oss-v1-0.6b endpoint deployed in Microsoft Foundry, encode all internal documents—policy guides, project case studies, and technical documentation—across English, French, German, and Japanese. At query time, prepend the task instruction to each employee query: "Instruct: Retrieve relevant internal documents that answer the employee's question\nQuery: {question}". Retrieve the top-5 most similar documents by cosine similarity and pass them to a language model with the instruction: "Using only the provided documents, answer the question and cite the source document title for each claim. If no document addresses the question, say so."

NVIDIA: EGM-8B

Model Specs

Parameters / size: ~8.8B
Context length: 262,144 tokens
Primary task: Image-text-to-text (visual grounding)

Why it's interesting

Preforms well on visual grounding compared to larger models even at its small size: EGM-8B achieves 91.4 average Intersection over Union (IoU) on the RefCOCO benchmark—the standard measure of how accurately a model localizes a described region within an image. Compared to its base model Qwen3-VL-8B-Thinking (87.8 IoU), EGM-8B achieves a +3.6 IoU gain through targeted Reinforcement Learning (RL) fine-tuning. Benchmarks: EGM Project Page.
5.9x faster than larger models at inference: EGM-8B achieves 737ms average latency. The research demonstrates that test-time compute can be scaled horizontally across small models—generating many medium-quality responses and selecting the best—rather than relying on a single expensive forward pass through a large model.
Two-stage training: EGM-8B is trained first with Supervised Fine-Tuning (SFT) on detailed chain-of-thought reasoning traces generated by a proprietary VLM, then refined with Group Relative Policy Optimization (GRPO) using a reward function combining IoU accuracy and task success. The intermediate SFT checkpoint is available as nvidia/EGM-8B-SFT for developers who want to experiment with the intermediate stage.
Addresses a root cause of small model grounding errors: The EGM research identifies that 62.8% of small model errors on visual grounding stem from complex multi-relational descriptions—where a model must reason about spatial relationships, attributes, and context simultaneously. By focusing test-time compute on reasoning through these complex prompts, EGM-8B closes the gap without increasing the underlying model size.

Try it

Use Case	Prompt Pattern
Object localization	Submit image + natural language description; receive bounding box coordinates
Document region extraction	Provide scanned document image + field description; extract specific regions
Visual quality control	Submit product image + defect description; localize defect region for downstream classification
Retail shelf analysis	Provide shelf image + product description; return location of specified SKU

Sample prompt for a retail and logistics deployment:

You are building a visual inspection system for a logistics warehouse. Using the EGM-8B endpoint deployed in Microsoft Foundry, submit each incoming package scan image along with a natural language grounding query describing the region of interest: "Please provide the bounding box coordinate of the region this sentence describes: {description}". For example: "the label on the upper-left side of the box", "the barcode on the bottom face", or "the damaged corner on the right side". Use the returned bounding box coordinates to route each package to the appropriate inspection station based on the identified region.

Getting started

You can deploy open-source Hugging Face models directly in Microsoft Foundry by browsing the Hugging Face collection in the Foundry model catalog and deploying to managed endpoints in just a few clicks. You can also start from the Hugging Face Hub. First, select any supported model and then choose "Deploy on Microsoft Foundry", which brings you straight into Azure with secure, scalable inference already configured. Learn how to discover models and deploy them using Microsoft Foundry documentation:

When Should You Use RAG vs Fine-Tuning in Microsoft Foundry?

SajedaSultana — Fri, 10 Apr 2026 01:23:43 GMT

If you've been working with Microsoft Foundry, you've likely come across this question:

Should I use RAG or Fine-Tuning?

The answer becomes much simpler when you focus on the core goal of your solution. Here's a straightforward way to think about it.

What is RAG (Retrieval-Augmented Generation)?

RAG allows your model to retrieve relevant information from your data sources before generating a response.

Instead of relying only on what the model already knows, it:

Searches your documents or knowledge base
Retrieves relevant content
Uses that context to generate grounded, cited answers

Use RAG when:

✅ Your data changes frequently

✅ You need answers based on real documents

✅ You have a large, evolving document library

✅ You are building "chat with your data" experiences

What is Fine-Tuning?

Fine-tuning customizes how the model behaves by training it on task-specific examples.

It helps the model:

Produce consistent and structured outputs
Follow a specific tone, format, or brand voice
Align with business rules, compliance policies, and workflows

Use Fine-Tuning when:

✅ You need consistent and predictable responses

✅ You want a specific tone, format, or behavior

✅ Your task is stable and well-defined

✅ You are operating at massive scale

Visual Overview

Below is a quick visual summary to help compare both approaches:

A Simple Way to Decide

Ask yourself:

Is my problem about accessing the right data, or controlling how the model behaves?

If it's about data → use RAG
If it's about behavior → use Fine-Tuning

Quick Comparison

What You Need	RAG	Fine-Tuning
Data changes often	✅ Yes	❌ Not ideal
Change model behavior/style	❌ No	✅ Yes
Fast to get started	✅ Faster	❌ Needs training
High-volume, stable queries	⚠️ Token costs grow	✅ Predictable cost
Brand voice / compliance	⚠️ Limited	✅ Built into model
Large, evolving document library	✅ Perfect fit	❌ Hard to maintain

Can You Use Both?

In many real-world scenarios, the best teams do exactly that:

RAG brings in the right, up-to-date information
Fine-Tuning ensures consistent behavior and output quality

Think of RAG as giving your model the right books to read, and Fine-Tuning as teaching it how to think and respond. Together, they cover both sides of the equation.

I'd love to hear from others in the community:

Are you using RAG, Fine-Tuning, or both in your Foundry projects?
What use cases are you solving?
What challenges or trade-offs have you experienced along the way?

Looking forward to your insights. Let's learn from each other! 🚀

Bringing GigaTIME to Microsoft Foundry: Unlocking Tumor Microenvironment Insights with Multimodal AI

Saumil-Shrivastava — Tue, 14 Apr 2026 11:36:56 GMT

Expanding Microsoft Foundry for Scientific AI Workloads

AI is increasingly being applied to model complex real-world systems, from climate and industrial processes to human biology. In healthcare, one of the biggest challenges is translating routinely available data into deeper biological insight at scale.

GigaTIME is now available in Microsoft Foundry, bringing advanced multimodal capabilities to healthcare and life sciences. This brings advanced multimodal capabilities into Foundry, with Foundry Labs enabling early exploration and the Foundry platform supporting scalable deployment aligned to the model’s intended use.

Read on to understand how GigaTIME works and how you can start exploring it in Foundry.

From Routine Slides to Deep Biological Insight

Understanding how tumors interact with the immune system is central to modern cancer research. While techniques like multiplex immunofluorescence provide these insights, they are expensive and difficult to scale across large patient populations.

GigaTIME addresses this challenge by translating widely available hematoxylin and eosin pathology slides into spatially resolved protein activation maps. This allows researchers to infer biological signals such as immune activity, tumor growth, and cellular interactions at a much deeper level. Developed in collaboration with Providence and the University of Washington, GigaTIME enables analysis of tumor microenvironments across diverse cancer types, helping accelerate discovery and improve understanding of disease biology.

Use cases for GigaTIME

GigaTIME is designed to support research and evaluation workflows across a range of real-world scientific scenarios:

Population-Scale Tumor Microenvironment Analysis: Enable large-scale analysis of tumor–immune interactions by generating virtual multiplex immunofluorescence outputs from routine pathology slides.
Biomarker Association Discovery: Identify relationships between protein activation patterns and clinical attributes such as mutations, biomarkers, and disease characteristics.
Patient Stratification and Cohort Analysis: Segment patient populations across cancer types and subtypes using spatial and combinatorial signals for research and hypothesis generation.
Clinical Trial Retrospective Analysis: Apply GigaTIME to H&E archives from completed clinical trials to retrospectively characterize tumor microenvironment features associated with treatment outcomes, enabling new insights from existing trial data without additional tissue processing.
Tumor–Immune Interaction Analysis: Assess whether immune cells are infiltrating tumor regions or being excluded by analyzing spatial relationships between tumor and immune signals.
Immune System Structure Characterization: Understand how immune cell populations are organized within tissue to evaluate coordination or fragmentation of immune response.
Immune Checkpoint Context Interpretation: Examine how immune activity may be locally regulated by analyzing overlap between immune markers and checkpoint signals.
Tumor Proliferation Analysis: Identify actively growing tumor regions by combining proliferation signals with tumor localization.
Stromal and Vascular Context Understanding: Analyze how tissue architecture, such as vascular density and desmoplastic stroma, shapes immune cell access to tumor regions, helping characterize mechanisms of immune exclusion or infiltration.

Start with Exploration, Then Go Deeper

Explore in Foundry Labs

Foundry Labs provides a lightweight environment for early exploration of emerging AI capabilities. It allows developers and researchers to quickly understand how models like GigaTIME behave before integrating them into production workflows.

With GigaTIME in Foundry Labs, you can engage with real-world healthcare scenarios and explore how multimodal models translate pathology data into meaningful biological insight.

Through curated experiences, you can:

Run inference on pathology images
Visualize spatial protein activation patterns
Explore tumor and immune interactions in context

This helps you build intuition and evaluate how the model can be applied to your specific use cases.

Go Deeper with GitHub Examples

For advanced scenarios, you can access the underlying notebooks and workflows via GitHub. These examples provide flexibility to customize pipelines, extend workflows, and integrate GigaTIME into broader research and application environments. Together, Foundry Labs and GitHub provide a path from guided exploration to deeper customization.

Discover and Deploy GigaTIME in Microsoft Foundry

Discover in the Foundry Catalog

GigaTIME is available in the Foundry model catalog alongside a growing set of domain-specific models across healthcare, geospatial intelligence, physical systems and more.

Deploy for Research Workflows

For advanced usage, GigaTIME can be deployed as an endpoint within Foundry to support research and evaluation workflows such as:

Biomarker discovery
Patient stratification
Clinical research pipelines

You can start with early exploration in Foundry Labs and transition to scalable deployment on the Foundry platform using the tools and workflows designed for each stage, in line with the intended use of the model.

A New Class of AI for Scientific Discovery

GigaTIME reflects a broader shift toward AI systems designed to model real-world phenomena. These systems are multimodal, deeply tied to domain-specific data, and designed to produce spatial and contextual outputs. They rely on workflows that combine data processing, model inference, and interpretation, which requires platforms that support the full lifecycle from exploration to production. Microsoft Foundry is built to support this evolution.

Learn More and Get Started

To explore GigaTIME in more detail:

Read the Microsoft Research blog on the underlying research, and population-scale findings.
Try hands-on scenarios in Foundry Labs
Access GitHub examples for advanced workflows
Explore and deploy the model through the Foundry catalog

Looking Ahead

As AI continues to expand into domains like healthcare, climate science, and industrial systems, the ability to connect models, data, and workflows becomes increasingly important. GigaTIME highlights what is possible when these elements come together, transforming routinely available data into actionable scientific insight.

We are excited to see what you build next.