Troubleshooting ML Model Loading, GPU Issues, and Memory Pressure in Azure Container Apps
Introduction Deploying an AI application to Azure Container Apps is fundamentally different from deploying a web API. When you containerize a Django REST API, the application starts in a few seconds, the memory footprint is predictable, and the CPU usage scales linearly with requests. When you containerize a PyTorch model server, a LangChain agent, or an ONNX inference service, you are dealing with a completely different category of problem. Large language models, computer vision models, and embedding pipelines can take minutes to load, consume gigabytes of memory before serving a single request, and produce bizarre errors when they encounter resource limits that look nothing like a standard out-of-memory exception. Add to that the challenge of running GPU workloads (or simulating them on CPU) in a containerized environment, and you have a troubleshooting landscape that catches even experienced ML engineers off guard. This part of the series covers the real-world scenarios you will encounter when running AI workloads on Azure Container Apps, with specific focus on what goes wrong during deployment and startup — and how to fix it methodically. Scenario 1: The Model Takes 3+ Minutes to Load and the Container Gets Killed Before It Starts What You See You deploy your model inference service. In the logs you can see it is downloading or loading the model from disk: INFO: Loading model from /app/models/model.bin INFO: Loading tokenizer... INFO: Loading weights layer 0/48... INFO: Loading weights layer 12/48... And then, abruptly, the container disappears and a new one starts. The liveness or startup probe has timed out and Container Apps has killed the container before the model finished loading. You end up in an endless restart loop where the model never fully loads. Why This Happens The default probe configuration does not account for long model loading times. The liveness probe begins firing almost immediately after the container starts. If your model takes 3 minutes to load and your liveness probe allows only 30 seconds of failure before killing the container, the model never gets a chance to finish loading. Container Apps is doing exactly what it was configured to do — it just was not configured with your workload in mind. There is a second, related problem: if your model file is downloaded at container startup (from Azure Blob Storage, Hugging Face Hub, or a mounted file share), the download time is added on top of the load time, making the window even wider. Step-by-Step Fix Step 1 — Separate your startup probe from your liveness probe. A startup probe fires repeatedly until it succeeds, and while it is in progress, the liveness probe is suppressed. This gives your model the time it needs to load without the risk of being killed. Set a generous `failureThreshold` and `periodSeconds`: probes: - type: Startup httpGet: path: /health/startup port: 8080 initialDelaySeconds: 10 periodSeconds: 15 failureThreshold: 40 # 40 * 15s = 600 seconds (10 minutes) for model to load - type: Liveness httpGet: path: /health/live port: 8080 periodSeconds: 30 failureThreshold: 3 - type: Readiness httpGet: path: /health/ready port: 8080 periodSeconds: 10 failureThreshold: 3 Step 2 — Implement a proper three-tier health endpoint in your model server. Each probe endpoint should return the appropriate status based on what it knows about the model: # FastAPI model server with staged health endpoints from fastapi import FastAPI, HTTPException from contextlib import asynccontextmanager import asyncio model = None model_loaded = False @asynccontextmanager async def lifespan(app: FastAPI): global model, model_loaded # Model loads in the background so the HTTP server starts immediately asyncio.create_task(load_model_async()) yield # Cleanup on shutdown model = None app = FastAPI(lifespan=lifespan) async def load_model_async(): global model, model_loaded import logging logger = logging.getLogger(__name__) logger.info("Starting model load...") try: # Import heavy libraries only when needed import torch from transformers import AutoModelForCausalLM, AutoTokenizer tokenizer = AutoTokenizer.from_pretrained("/app/models/my-model") model_obj = AutoModelForCausalLM.from_pretrained( "/app/models/my-model", torch_dtype=torch.float16, device_map="auto" ) model = {"tokenizer": tokenizer, "model": model_obj} model_loaded = True logger.info("Model loaded successfully") except Exception as e: logger.error(f"Model loading failed: {e}", exc_info=True) raise @app.get("/health/startup") def startup_probe(): # Returns 200 immediately — the container is alive but model may still be loading return {"status": "starting"} @app.get("/health/live") def liveness_probe(): # Returns 200 as long as the process has not entered a broken state return {"status": "alive"} @app.get("/health/ready") def readiness_probe(): # Returns 200 ONLY when the model is fully loaded if not model_loaded: raise HTTPException(status_code=503, detail="Model not yet loaded") return {"status": "ready"} Step 3 — Pre-download model weights into the container image at build time. Downloading models at container startup is a significant reliability and performance risk. If the Hugging Face Hub or your storage account is temporarily unreachable, the container cannot start. Instead, bake the model weights directly into the image or use a separate initialization Container App Job to pre-populate a persistent volume: # Option A: Bake the model into the image (simple, but creates a large image) FROM python:3.11-slim WORKDIR /app RUN pip install transformers torch --index-url https://download.pytorch.org/whl/cpu # Download model at build time RUN python -c "from transformers import AutoTokenizer, AutoModel; AutoTokenizer.from_pretrained('sentence-transformers/all-MiniLM-L6-v2', cache_dir='/app/models'); AutoModel.from_pretrained('sentence-transformers/all-MiniLM-L6-v2', cache_dir='/app/models')" COPY . . CMD ["uvicorn", "main:app", "--host", "0.0.0.0", "--port", "8080"] # Option B: Pre-populate an Azure Files volume using a Container App Job az containerapp job create --name model-downloader-job --resource-group my-rg --environment my-aca-env --trigger-type Manual --replica-timeout 600 --image python:3.11-slim --command "bash" --args "-c" "pip install huggingface_hub && huggingface-cli download my-org/my-model --local-dir /mnt/models" --volume-mounts "model-storage:/mnt/models" --volumes "model-storage:azureFile:my-fileshare" Scenario 2: The Model Server Runs Out of Memory and Gets OOM-Killed What You See Your model server starts successfully in development with 8 GB of RAM. In Azure Container Apps with a 4.0 vCPU / 8.0 Gi configuration (the maximum without GPU support), it crashes intermittently. The container restarts with no error in the application logs. When you check the system logs, you see the container exit code is `137`. Exit code 137 indicates the process was killed by the kernel OOM killer. Or in Log Analytics: ContainerAppSystemLogs_CL | where ContainerAppName_s == "my-model-server" | where Log_s contains "OOMKilled" or Log_s contains "137" | project TimeGenerated, Log_s Why This Happens Exit code 137 means the container was sent SIGKILL (`128 + 9 = 137`) because it exceeded its memory limit. Container Apps enforces memory limits strictly. When the process tries to allocate more memory than the container is allowed, the Linux kernel's OOM (Out Of Memory) killer terminates the process. With ML models, memory usage is not constant. A model might use 4 GB at rest but spike to 7 GB during inference when the input batch is large, when attention maps are being computed, or when the model is warming up its KV cache. If your container limit is 8 Gi and the model uses 7 Gi at rest plus 2 Gi during inference, you will hit OOM under load even if the container "usually" has enough memory. Step-by-Step Fix Step 1 — Quantize your model to reduce its memory footprint. Full-precision (FP32) models use twice the memory of half-precision (FP16) models, and INT8 quantized models use half the memory of FP16. For inference workloads where a small accuracy trade-off is acceptable, quantization is the single most impactful optimization you can make: import torch from transformers import AutoModelForCausalLM, AutoTokenizer, BitsAndBytesConfig # 4-bit quantization — reduces a 7B parameter model from ~14GB to ~4GB quantization_config = BitsAndBytesConfig( load_in_4bit=True, bnb_4bit_compute_dtype=torch.float16, bnb_4bit_quant_type="nf4", bnb_4bit_use_double_quant=True ) model = AutoModelForCausalLM.from_pretrained( "my-model-path", quantization_config=quantization_config, device_map="auto" ) Step 2 — Implement request batching to flatten memory spikes. Processing one request at a time with a large model creates memory spikes every time a new request arrives. Batching requests together means the model stays in a steady memory state rather than spiking per request: from asyncio import Queue, wait_for import asyncio class BatchedInferenceService: def __init__(self, model, tokenizer, max_batch_size=4, timeout=0.05): self.model = model self.tokenizer = tokenizer self.max_batch_size = max_batch_size self.timeout = timeout self.queue = Queue() async def infer(self, text: str) -> str: future = asyncio.get_event_loop().create_future() await self.queue.put((text, future)) return await future async def process_batches(self): while True: batch = [] futures = [] # Wait for the first item text, future = await self.queue.get() batch.append(text) futures.append(future) # Try to collect more items within the timeout window try: while len(batch) < self.max_batch_size: text, future = await wait_for(self.queue.get(), timeout=self.timeout) batch.append(text) futures.append(future) except asyncio.TimeoutError: pass # Process the whole batch at once try: results = self._run_inference_batch(batch) for future, result in zip(futures, results): future.set_result(result) except Exception as e: for future in futures: future.set_exception(e) def _run_inference_batch(self, texts): with torch.no_grad(): inputs = self.tokenizer(texts, return_tensors="pt", padding=True, truncation=True) outputs = self.model(**inputs) return outputs.logits.tolist() Step 3 — Set memory limits explicitly in your Container App to match what you expect at peak. Do not let the container use all available node memory and get killed unexpectedly. Set explicit limits that match your model's peak usage: # For a model that uses 5.5 GB at peak, allocate 6 GB az containerapp update --name my-model-server --resource-group my-rg --cpu 2.0 --memory 4.0Gi Step 4 — Add memory monitoring to your health endpoint. Your readiness probe should check available memory and return 503 if memory pressure is too high, which will cause the load balancer to route traffic to other healthy replicas: import psutil @app.get("/health/ready") def readiness_probe(): if not model_loaded: raise HTTPException(status_code=503, detail="Model not yet loaded") memory = psutil.virtual_memory() if memory.percent > 90: raise HTTPException( status_code=503, detail=f"Memory pressure too high: {memory.percent:.1f}% used" ) return { "status": "ready", "memory_percent": memory.percent, "available_gb": memory.available / (1024**3) } Scenario 3: GPU Workloads Fail to Initialize What You See You deploy a container that uses PyTorch with CUDA support. The container starts, but you see errors like: CUDA error: no kernel image is available for execution on the device torch.cuda.is_available() returned False RuntimeError: CUDA out of memory. Tried to allocate 2.00 GiB Or the model silently falls back to CPU without telling you, and your inference times are 50x slower than expected. Why This Happens Azure Container Apps supports GPU workloads through a dedicated GPU consumption plan and specialized GPU-enabled environments. If you deploy a CUDA-enabled container to a standard Container Apps environment (which uses CPU-only nodes), `torch.cuda.is_available()` returns `False` and PyTorch either errors out or silently falls back to CPU depending on how your code handles it. Even when you are in a GPU-enabled environment, CUDA version mismatches between the CUDA toolkit installed in your container image and the CUDA drivers on the host node will produce the "no kernel image" error. Step-by-Step Fix Step 1 — Detect whether GPU is available and log it explicitly at startup. Never assume — always log the device your model is using: import torch import logging logger = logging.getLogger(__name__) def initialize_device(): if torch.cuda.is_available(): device = torch.device("cuda") logger.info(f"Using GPU: {torch.cuda.get_device_name(0)}") logger.info(f"CUDA version: {torch.version.cuda}") logger.info(f"Available GPU memory: {torch.cuda.get_device_properties(0).total_memory / 1e9:.1f} GB") else: device = torch.device("cpu") logger.warning("GPU not available, falling back to CPU. Inference will be slower.") return device device = initialize_device() model = model.to(device) Step 2 — Match your CUDA toolkit version to the host driver. The CUDA toolkit version in your image must be less than or equal to the CUDA driver version on the host. Use the official PyTorch images which bundle compatible CUDA versions: # Use PyTorch's official image with a specific CUDA version # Check Azure Container Apps GPU documentation for supported CUDA versions FROM pytorch/pytorch:2.1.0-cuda11.8-cudnn8-runtime WORKDIR /app COPY requirements.txt . RUN pip install --no-cache-dir -r requirements.txt COPY . . CMD ["uvicorn", "main:app", "--host", "0.0.0.0", "--port", "8080"] Step 3 — Create your Container Apps environment with GPU support enabled. GPU support requires a dedicated GPU workload profile in your Container Apps environment: # Create an environment with GPU workload profil az containerapp env create --name my-gpu-env --resource-group my-rg --location eastus --workload-profile-type "NC24-A100" --workload-profile-name "gpu-profile" # Deploy your app to the GPU profile az containerapp create --name my-model-server --resource-group my-rg --environment my-gpu-env --image myregistry.azurecr.io/my-model-server:cuda11.8 --cpu 4.0 --memory 16.0Gi --workload-profile-name "gpu-profile" --min-replicas 1 Step 4 — Handle the CPU fallback gracefully so you know when it happens. If you want your application to work on both GPU and CPU environments, implement a clean fallback that makes the degraded state visible in monitoring: import os import torch class ModelConfig: def __init__(self): self.force_cpu = os.environ.get("FORCE_CPU", "false").lower() == "true" self.device = self._select_device() def _select_device(self): if self.force_cpu: return torch.device("cpu") if torch.cuda.is_available(): return torch.device("cuda") # GPU was expected but not found — emit a warning metric import warnings warnings.warn( "CUDA requested but not available. Running on CPU. " "Inference latency will be significantly higher.", RuntimeWarning ) # Optionally: push a custom metric to Azure Monitor here return torch.device("cpu") @property def is_gpu(self): return self.device.type == "cuda" Scenario 4: LangChain / AI Agent Timeouts at Startup What You See You deploy a LangChain-based agent or a RAG (Retrieval-Augmented Generation) pipeline to Container Apps. During startup, the application connects to Azure OpenAI, loads an embedding model, and populates an in-memory vector store. But the readiness probe times out while the vector store is being populated from a large document set. Why This Happens LangChain applications often do expensive work at startup — embedding thousands of documents, pre-populating vector indexes, or loading conversation history from a database. This work happens synchronously in many LangChain components, blocking the main thread and preventing the HTTP server from responding to health probes. Step-by-Step Fix Step 1 — Move initialization work to a background task that runs after the HTTP server is up from fastapi import FastAPI from contextlib import asynccontextmanager import asyncio import logging logger = logging.getLogger(__name__) vector_store = None initialization_complete = False initialization_error = None @asynccontextmanager async def lifespan(app: FastAPI): global initialization_complete, initialization_error # Start the heavy initialization in the background task = asyncio.create_task(initialize_vector_store()) yield # Server starts and begins serving health probe requests # Cleanup task.cancel() app = FastAPI(lifespan=lifespan) async def initialize_vector_store(): global vector_store, initialization_complete, initialization_error try: logger.info("Starting vector store initialization...") # Run CPU-bound work in a thread pool so we don't block the event loop loop = asyncio.get_event_loop() vector_store = await loop.run_in_executor(None, _build_vector_store) initialization_complete = True logger.info("Vector store initialization complete") except Exception as e: initialization_error = str(e) logger.error(f"Vector store initialization failed: {e}", exc_info=True) def _build_vector_store(): from langchain_community.vectorstores import FAISS from langchain_openai import AzureOpenAIEmbeddings from langchain_community.document_loaders import DirectoryLoader loader = DirectoryLoader("/app/documents") documents = loader.load() embeddings = AzureOpenAIEmbeddings( azure_deployment=os.environ["AZURE_OPENAI_EMBEDDING_DEPLOYMENT"], azure_endpoint=os.environ["AZURE_OPENAI_ENDPOINT"], api_key=os.environ["AZURE_OPENAI_API_KEY"] ) return FAISS.from_documents(documents, embeddings) @app.get("/health/ready") def readiness(): if initialization_error: raise HTTPException(status_code=500, detail=f"Initialization failed: {initialization_error}") if not initialization_complete: raise HTTPException(status_code=503, detail="Initializing vector store, please wait...") return {"status": "ready"} Step 2 — Use Azure AI Search instead of an in-memory vector store for large document sets. In-memory vector stores like FAISS are fine for development but become a liability in production Container Apps because the index is lost every time the container restarts, and rebuilding it adds minutes to your startup time. Azure AI Search persists the index and provides near-instant load times: from langchain_community.vectorstores.azuresearch import AzureSearch from langchain_openai import AzureOpenAIEmbeddings embeddings = AzureOpenAIEmbeddings( azure_deployment=os.environ["AZURE_OPENAI_EMBEDDING_DEPLOYMENT"], azure_endpoint=os.environ["AZURE_OPENAI_ENDPOINT"], api_key=os.environ["AZURE_OPENAI_API_KEY"] ) # The index already exists in Azure AI Search — no rebuild needed on startup vector_store = AzureSearch( azure_search_endpoint=os.environ["AZURE_SEARCH_ENDPOINT"], azure_search_key=os.environ["AZURE_SEARCH_KEY"], index_name="my-document-index", embedding_function=embeddings.embed_query ) # This is nearly instantaneous — just connects to the existing index initialization_complete = True Debugging AI Workload Logs in Log Analytics When something goes wrong with your AI workload, these Log Analytics queries will help you quickly identify the pattern: // Find all OOM kills in the last 24 hours ContainerAppSystemLogs_CL | where TimeGenerated > ago(24h) | where Log_s contains "OOMKilled" or ExitCode_d == 137 | project TimeGenerated, ContainerAppName_s, Log_s, ExitCode_d | order by TimeGenerated desc // Track model loading time across restarts ContainerAppConsoleLogs_CL | where ContainerAppName_s == "my-model-server" | where Log_s contains "Model loaded" or Log_s contains "Starting model load" | project TimeGenerated, Log_s, ContainerName_s | order by TimeGenerated asc // Find slow inference requests (if you are logging inference latency) ContainerAppConsoleLogs_CL | where ContainerAppName_s == "my-model-server" | where Log_s contains "inference_latency_ms" | extend latency = toint(extract("inference_latency_ms=([0-9]+)", 1, Log_s)) | where latency > 5000 // Requests taking more than 5 seconds | project TimeGenerated, latency, Log_s | order by latency desc Summary: AI Workload Startup Checklist When your AI workload fails to start or behaves unexpectedly, work through this checklist: - Is the model loading time covered by an appropriate startup probe `failureThreshold`? - Are model weights baked into the image or pre-loaded onto a mounted volume, rather than downloaded at runtime? - Is the container's memory limit large enough for peak inference load (model size + activation memory)? - Have you verified whether the workload is running on GPU or CPU and logged that explicitly at startup? - Does the CUDA version in your image match or precede the driver version on the host? - For LangChain or RAG workloads, does initialization happen in the background so health probes can respond? - Are you using a persistent vector store (Azure AI Search, Azure Cosmos DB) instead of rebuilding in-memory indexes on every restart? References and Sample Resources Use these links alongside the scenarios above to go deeper on configuration details and production patterns. Azure Container Apps docs (core) Health probes in Azure Container Apps Workload profiles overview (Consumption, Dedicated, Memory-optimized, GPU) Serverless GPU overview and supported regions Azure Container Apps Jobs (for preloading models or maintenance tasks) Monitoring and logging options in Azure Container Apps AI-specific docs AI integration with Azure Container Apps LangChain tutorial with Container Apps dynamic sessions Azure AI Search vector search overview Azure OpenAI with Python quickstart Official sample repositories Chat app with Azure OpenAI on Container Apps AI MCP server sample on Container Apps .NET MCP agent app (client/server pattern on Container Apps) TypeScript MCP container sample Python code interpreter dynamic session sample ML runtime and model-loading references PyTorch CUDA notes and best practices Hugging Face Transformers loading models from local/Hub paths bitsandbytes quantization guide What's Next In the final part of this series, we step back from reactive troubleshooting and look at how to build a proactive observability and automation layer so that you catch these problems before your users do — or better yet, have them resolve themselves automatically. Part of the series: Troubleshooting Azure Container Apps in Production Next: Part 4 — Eyes Open, Hands Free: Automating Observability, Alerts, and Self-Healing Diagnostics for Azure Container Apps🎉 Automation just became a team sport. Meet Azure Logic Apps Automation.
Low barrier to entry. Built for production. Now in Public Preview There's a moment that plays out in almost every organization right now. Someone closest a business problem - a retail ops lead, a finance analyst, a security analyst looks at a repetitive process and thinks, this should just run itself. For most of computing history, turning that idea into reality required specialized skills, significant setup, and engineering resources that were often focused elsewhere. AI is changing that. Today, people can describe what they want in natural language and watch working solutions take shape. The bottleneck is no longer generating an idea for automation. It's turning that idea into something secure, governed, and reliable enough to run in production. The demos are everywhere. The question organizations are increasingly asking is the harder one: which of these can we actually run in production? That's exactly the shift we built for. Today at Microsoft Build we're introducing Azure Logic Apps Automation, a new Logic Apps SKU that delivers the experience of a modern SaaS product for creating and running workflow automations. It makes it easier for teams to get started quickly while preserving the security, governance, reliability, and scale organizations expect from Azure. It's open to builders of every kind, available now in public preview at https://auto.azure.com. New experience, same enterprise engine The goal was straightforward: simplify the experience of building and running automations without compromising the enterprise foundation underneath. Logic Apps Automation provides a managed experience where compute, model endpoints, knowledge services, and execution environments are available out of the box. Teams can focus on solving business problems rather than assembling infrastructure and services. We also introduced a dedicated SaaS experience designed around productivity and collaboration. Administrators establish governance and policies, while builders can quickly begin creating workflows without requiring deep Azure expertise. "The redesigned experience lets me build AI-based solutions in record time. This platform will serve as the glue in most modern solutions.", Mick Badran, Founder & Director at SolveIT.Today [LA Automation Early Adopter] What we kept is just as important. Logic Apps Automation is built on the same Azure Logic Apps platform organizations trust today. The reliability, scale, security, governance, and operational maturity remain the foundation. The experience is simpler, but the platform underneath is the same proven technology customers rely on every day. Low barrier to entry. Built for production. We mean both halves of that sentence. Build like a startup, ship like an enterprise Building an automation is only part of the full application journey. As solutions move from experimentation to production, along with simple experience, organizations need security, governance, networking, identity, and operational controls to ensure those automations can be trusted at scale.Logic Apps Automation is designed for both realities. On the build side, it's fast to get started. Login and start building workflows; stay on a single canvas throughout the experience: use AI assisted workflow development, use visual workflows when they’re the right fit, and drop into code the moment you need additional control. No switching tools, no handoffs, no separate infrastructure to manage. On the production side, organizations get the capabilities they expect from an enterprise platform, on day-0: isolated compute, virtual network integration and private endpoints, identity, role-based access, audit logging, and governance policies. For many automation tools, becoming "enterprise-ready" is something that happens later. With Logic Apps Automation , production-readiness is part of the foundation. Built for how teams actually work Making automation easier for builders shouldn't create additional complexity for administrators. Organizations already have established governance boundaries, ownership models, and operational processes. Logic Apps Automation is designed to align with those realities through a simple two-level hierarchy of Projects and Applications. Project sits at the top and act as your security and governance boundary; inside each project you run one or more Applications. Admins and project owners set networking policies, connector policies, sandbox configuration, and approved AI models once, at the project scope and every application inherits them. Builders get a wide-open space to create. Admins get a firm line around it. Nobody has to choose between the two. Flexible permission management for individuals and teams The permission model is also designed to match how teams collaborate: A private space for an individual. To give a single user a place to run their own automations with a privacy boundary around personal resources such as their email account - create an application that only that individual can access. A shared space for a team. To support an automation that several people co-develop and operate together, add multiple users to the application so they can build, run, and maintain it collectively. The same model accommodates both access patterns, giving builders clear control over the scope of each application and who can work within it. AI-native, not AI-retrofitted Logic Apps Automation is designed for a new generation of business processes that combine workflows, AI agents, enterprise systems, and human decision-making. It starts with how you build. A built-in AI Assistant turns plain language into working automation. You describe what you want and it drafts the workflow, configures actions, writes expressions, and generates inline code, then helps you edit the same way. You can author at the level of a single step or an entire end-to-end flow. This is the thing that opens the platform to *every* developer: the person closest to the problem can describe it and get something real, while pros stay in control and drop to code whenever they want. "With the power of AI, automations just got on steroids! Simply tell it what you need, explain the intent, et voilà! Love it.", Sonny Gillissen, Integration Architect at Rubicon Cloud Advisor [LA Automation Early Adopter] Agents are first-class Agents are first-class, and we meet you where you are with three ways to integrate them: Agent-loop orchestration. If you're already using Logic Apps actions as tools inside an agent loop, that pattern carries forward. Your actions are callable tools the agent can invoke, so you keep orchestrating the way you always have. Foundry agents. Connect to an existing Microsoft Foundry Hosted or Prompt Agent or create a new one right from the canvas. The platform handles the wiring, and your workflow calls the agent, gets results back, and keeps moving. Managed sandbox for agent harnesses. Bring a well-known agent harness, like GitHub Copilot and run it in a managed, isolated sandbox. We take care of the compute, the isolation, native shell access, and your GitHub repos as first-class context; you just define the business logic. Then orchestrate all of these inside a larger workflow, right next to traditional rule-based actions, on a single canvas. Deterministic and agentic, in one place. A few capabilities that make this especially powerful: Sandboxed agent harnesses. Run agent harnesses such as GitHub Copilot in a managed, isolated sandbox with shell execution, skills, and GitHub repos as first-class context, without operating any of that infrastructure yourself. Tools and MCP. Turn any of the 1400+ connectors into a tool or expose any workflow as an MCP server that any compatible agent can call. No code required. Knowledge as a Service. Drop in your documents and the platform handles ingestion, chunking, embeddings, and retrieval. No RAG pipeline to build, no vector store to operate; just grounded answers. Any model, anywhere. Plug in whatever fits the job: frontier, open-source, fine-tuned, or local. You're never locked in. "Azure Automation closes the gap between integration and intelligence with agents as first-class workflow actions, grounded in your own data, executing in isolated sandboxes, all within the same canvas where your triggers and connectors live. Excited to see the evolution.", Sagar Sharma, Enterprise Solution Architect at i8c NL [LA Automation Early Adopter] What's new in this release Logic Apps Automation introduces several new capabilities designed to help teams build, deploy, and govern AI-powered automations: Zero-friction onboarding. Get from Sign-in to first workflow in minutes, with managed infrastructure and enterprise capabilities available from the start. A new designer. Modern designer with single pane experience to build and monitor workflows, draft-mode for workflows for easy iterations, instant code-to-workflow synchronization when you want to work in code-view, run history you can stream live, and so much more Natural language authoring. Describe workflows in plain language to create and edit them, with AI assistance in the designer. More powerful agents. Three ways to bring agents into a workflow; agent-loop orchestration, Foundry Hosted Agents, and well-known harnesses like GitHub Copilot running in a managed, isolated sandbox with shell access and GitHub repos as context. Knowledge as a Service. A managed knowledge layer that turns your documents into a ready-to-use knowledge base; no RAG pipeline required. JavaScript expressions. Write inline JavaScript to transform data and express logic without leaving the designer; no domain-specific language to learn. Projects and applications. A two-level governance hierarchy that gives admins a clear boundary and builders room to create. A permission management model that accommodates different level of access patterns, giving builders clear control over the scope of each application and who can work within it. Elastic scale, including to zero. Workflows scale up automatically when load arrives and scale all the way down to zero when there's no work to do. You pay only for the vCPU-seconds you actually use. Built to scale Logic Apps Automation scales automatically with demand, from idle workloads to business-critical processes. Customers pay only for the resources they use, without per-seat licensing requirements or infrastructure management overhead. When workflows aren't running, you're not paying for compute. When demand increases, the platform scales with you. Pricing Logic Apps Automation uses a consumption-based pricing model, so you pay only for what you use. Pricing is based on a small managed-environment fee, workflow execution, and optional services such as AI model usage, knowledge, sandboxes, connector calls. There is no annual commitment, no per-seat license, no quota cliff. When your workflows sit idle, you pay nothing for compute. More details to follow soon. What's available, and what's next Logic Apps Automation is available today in public preview, with an intial set of regions today, with more rolling out over the coming weeks. Here is the list of regions its available today: East Asia Sweden Central Australia East North Central US UK South Southeast Asia West US Coming Soon We're continuing to expand the platform with additional AI and enterprise capabilities, including: Foundry Hosted Agents. Create or Invoke Foundry Hosted Agents directly inside your workflows. Foundry Prompt Agents. Create/Invoke Foundry prompt directly inside your workflows. Hosted Models. Managed model endpoints provided for you; no keys or infrastructure to bring. Inline Python. Write inline Python alongside JavaScript when you need it. Bring your own container image. Run your own code in sandboxes; for example, orchestrate a Python ETL job from within a Logic Apps workflow. VNet support and private endpoints. Custom connectors and more Automation templates. Build custom connectors, start from a growing library of templates, and set project-level policies on connectors and more. Get started Whether you're automating a business process, orchestrating AI agents, integrating enterprise systems, or building entirely new AI-powered experiences, Logic Apps Automation provides a simpler path from idea to production. Start building today at https://auto.azure.com Read the docs at http://auto.azure.com/docs Watch the announcement session at Microsoft Build 2026. See it live at the Integrate conference, June 8–9.Design observability for AI apps and agents selling through Microsoft Marketplace
In the last post, API resilience and reliability patterns for AI apps and agents, we focused on what happens when AI systems encounter failure—and how resilient execution paths keep that failure contained. Timeouts fire with intent. Retries stay bounded. Circuit breakers provide overload protection. When resilience is designed well, your system continues to function even as conditions change, forming the foundation of AI reliability engineering. You can always get curated step-by-step guidance through building, publishing and selling apps for Marketplace through App Advisor. This post is part of a series on building and publishing well-architected AI apps and agents in Microsoft Marketplace. The series focuses on AI apps and agents that are architected, hosted, and operated on Azure, with guidance aligned to building and selling solutions through Microsoft Marketplace. Observability for AI systems AI apps and agents are shifting traditional observability, which was designed for systems based on simple assumptions, where requests followed linear paths and workloads behaved predictably. Execution in AI systems consumes tokens at a highly variable rate rather than fixed compute units. Requests unfold across multiple reasoning steps. Agents perform work that spans APIs, models, retrieval layers, and applications. A single interaction may pause, branch, retry, or exit early depending on inferred intent, context, and constraints. Instead of asking whether services are running, observability for AI systems asks: what is the system doing right now—and why? Is an agent spending its time reasoning, waiting on dependencies, retrying tool calls, or exiting early due to enforced limits? Is cost increasing because value is increasing, or because execution paths are expanding without progress? AI observability requirements shift the focus in the following subtle, but critical ways: From resource availability to workflow state From performance metrics to signals From incidents to patterns Core observability dimensions for AI apps and agents Once observability shifts toward understanding behavior, clarity comes from tracking state across the agents in the workflow. For AI apps and agents, observable indicators, such as those detailed below, show how work unfolds and changes during real usage—especially in trials and early adoption: Execution flow shows how a request moves through agents, tools, and workflows. This highlights where execution progresses smoothly, where it slows, and where it concludes early. This makes agent outcomes explainable and keeps behavior consistent across tenants. Cost and token behavior reveals how execution translates into consumption. Token usage per request, per agent step, and per retry shows where value is being delivered and where execution paths expand without proportional benefit. This insight connects runtime behavior directly to Marketplace billing expectations and evaluations. Latency and wait states distinguish active processing from time spent waiting on dependencies. Seeing where time is consumed helps explain slow experiences and guides decisions about optimization, caching, or resilience improvements. Failure classification provides structure when systems degrade and supports effective AI incident management. Separating tool failures from planning failures, and transient issues from terminal exits, keeps investigations focused and prevents protective behavior from being misread as instability. Tenant‑level patterns surface how behavior repeats at scale. Uneven load, and recurring degradation often appear first during trials and shape the customer's perception. Together, these dimensions turn telemetry into understanding—supporting clearer conversations, faster triage, and predictable execution as usage grows. Why observability matters By this point in the journey, your AI app or agent has implemented bounded execution paths, cost controls, and quality of service safeguards. As a result, failure degrades gracefully instead of spreading. These resilience techniques determine how your solution behaves under pressure. The data gathered from observability platforms like Application Insights and Azure Monitor explains why it behaves that way. For AI and agentic systems, infrastructure health alone rarely answers the questions that matter. Services can be up, CPUs can be idle, and queues can look healthy while agents loop inefficiently, retries quietly expand cost, or workflows exit early without delivering value. From the customer’s perspective, the experience feels inconsistent even though the platform appears stable. AI app observability closes this gap by revealing system behavior rather than system status. It shows how requests move, where work concentrates, and how constraints shape outcomes. At Marketplace scale, these patterns repeat across tenants and trials. What appears once during an evaluation often appears again as adoption grows. Observability connects runtime behavior back to the design choices introduced in earlier posts: Usage‑based billing introduced variability in consumption Performance optimization introduced tradeoffs among latency, quality, and cost Resilience patterns introduced controlled failure and bounded execution Observability allows you to explain outcomes during trials, validate assumptions as usage grows, and support post-launch AI operations confidence across customers and environments. Without this visibility, teams react to symptoms. With it, they recognize patterns. From execution paths to behavioral signals Observability begins at the same place resilience begins—API boundaries. These boundaries define where responsibility shifts and where behavior becomes visible. Observability focuses on signals that explain decisions made by the system as it executes instead of relying on raw logs that describe isolated events. Every resilience mechanism emits behavioral signals. Viewed together, these signals provide far more value than logs alone. Logs answer whether something happened. Behavioral signals explain why it happened and how the system responded. Circuit breakers change state as load builds and recedes. Retry loops show whether failures resolve quickly or exhaust their limits. Timeout enforcement reveals where dependencies slow execution. Fallback paths and early terminations show how the system protects itself while preserving outcomes for customers. This perspective matters most for agents. Agent execution unfolds as a series of choices—plan, call a tool, retry, exit early—rather than a single request‑response cycle, which requires monitoring AI agent behavior to remain understandable and consistent at scale. Observability that tracks these decisions makes agent behavior understandable, consistent, and defensible as usage grows across customer tenants. Observability at the agent layer As AI systems become more agent‑driven, observability needs to move closer to where decisions are made. Agents introduce variability by design. They plan, adapt, and choose workflow paths dynamically. Without first‑class visibility into that behavior, execution can appear unpredictable even when the underlying system is healthy. Observability at the agent layer acts as the feedback loop that keeps execution safely bounded. It shows how agents use the freedom you give them—and where that freedom begins to stretch into inefficiency. Observability follows how the agent did its job instead of treating the agent’s interaction as a single outcome. Several indicators help make agent behavior understandable. Step count per request reveals how much reasoning effort a prompt requires. Planning iterations show whether an agent converges quickly or cycles through alternatives. Tool invocation frequency highlights when agents rely heavily on external systems. Early exits compared to full completion explain whether limits and fallbacks activate as designed. Taken together, these indicators help distinguish healthy exploration from inefficient reasoning and degraded execution. An agent exploring briefly before converging adds value. An agent looping through tools without progress signals pressure, uncertainty, or dependency issues. This distinction reinforces a core principle of agentic systems: models reason probabilistically, adapting to context as it changes. Your system observes deterministically—measuring execution, enforcing boundaries, and clarifying outcomes. When those roles stay separate and well‑instrumented, agent behavior becomes transparent, predictable, and ready for Marketplace scale. Observability across environments The type of Marketplace offer you choose shapes what observability customers expect and how responsibility is shared. For SaaS offers, publishers typically own end‑to‑end execution. Observability centers on agent behavior, workflow completion, token usage, latency, and dependency impact across tenants. Publishers rely on consistent signals—often surfaced through tools like Azure Monitor, Application Insights, and Microsoft AI Foundry—to explain how requests behave as scale and load increase. For container‑based offers and Azure Managed Applications, observability expectations are more distributed. Publishers expose clear execution outcomes, limits, and failure signals at application boundaries. Customers, in turn, observe infrastructure health, scaling behavior, and downstream systems within their own environments. This separation ensures each party has visibility into what they control without creating ambiguity. Learn more about Choosing your marketplace offer type for AI Apps and agents. Execution behavior differs across environments for predictable reasons. Scale increases, tenant mix broadens, and external dependencies behave differently under real load. What must stay consistent is how behavior is interpreted. Signal definitions, thresholds, and failure classification should mean the same thing in Dev, Stage, and Prod. Learn more about designing a reliable environment strategy for Microsoft Marketplace AI apps and agents. Staging environments are where this consistency is validated. Observing retries, timeouts, and graceful degradation before production prepares you for Marketplace evaluations, which often resemble production conditions. Observability gaps tend to appear first during customer evaluation—when clarity matters most. Publisher and customer visibility boundaries Purpose: Parallel Post #13 responsibility clarity, now for observability As observability matures across environments, clarity around responsibility becomes essential. For Marketplace solutions, trust grows when publishers and customers each see what they own—and understand where that visibility ends. Publishers are responsible for instrumenting execution paths end to end. That means making workflows traceable, limits visible, and failure modes explainable. Observability should surface behavior—how requests progressed, where execution concluded, and why—rather than exposing raw internal errors that require insider knowledge to interpret. Customers focus their observability on what they control. This includes monitoring downstream systems, infrastructure behavior, and environment‑level alerts within their own estate. When visibility aligns with ownership, teams can act quickly and decisively. Exposing too much internal detail can overwhelm customers and blur accountability. Observing too little behavior creates friction, especially when issues cross boundaries and lack context. Clear visibility enables faster triage, sharper ownership boundaries, and fewer escalations rooted in ambiguity. Observability as an enabler for scale, billing, and trust From a customer’s perspective, observability answers two fundamental questions: Can I understand what happened? and Can I trust this at scale? When the answer to both is clear, observability becomes part of the value your Marketplace offering delivers. When system behavior is visible and explainable, customers gain confidence that adoption and growth will remain predictable. Observability directly supports usage‑based billing by tying execution behavior to measured consumption. Clear visibility into token usage, retries, and execution paths helps validate how usage is calculated and supports transparent billing conversations. It also enables ongoing performance tuning and caching strategies by showing where latency accumulates, where work repeats, and where optimization delivers measurable impact. Observability reinforces confidence in resilience mechanisms, confirming that limits, fallbacks, and degradation paths activate as designed under real‑world conditions. Beyond validation, observability creates a continuous feedback loop. Execution data informs pricing adjustments, guides changes to limits, and helps refine default configurations as customer behavior evolves. What’s next in the journey With execution behavior observable and explainable, the focus shifts to how AI systems are operated safely as change accelerates. The upcoming posts will discuss deployment strategies, CI/CD pipelines for agents, and progressive rollouts build on this foundation—ensuring AI apps evolve confidently as usage and expectations grow. Key Resources See curated, step-by-step guidance to help you build, publish, or sell your app or agent (no matter where you start) in App Advisor Quick-Start Development Toolkit can connect you with code templates for AI solution patterns Microsoft AI Envisioning Day Events How to build and publish AI apps and agents for Microsoft Marketplace Get over $126K USD in benefits and technical consultations to help you replicate and publish your app with ISV SuccessPartner Blog | Building AI-ready applications on open, enterprise-grade Azure platforms
Microsoft Build 2026 reinforced a practical reality for organizations moving from AI experimentation to production: AI is only as strong as the foundation it runs on. Customers need modern databases, governed data, secure infrastructure, and developer experiences that can carry performing AI-enabled applications at scale into production with confidence. The public preview announcements of Azure HorizonDB and Azure Linux, along with the general availability of Azure Container Linux, show how Microsoft is investing in open platforms, developer ecosystems, and enterprise-grade cloud infrastructure for AI-ready applications. These announcements also point to a broader shift: open source has become a strategic foundation for enterprise modernization, innovation, and strategic growth. These announcements give partners a straightforward way to connect customer AI goals to the open, secure, enterprise-ready platforms needed for production. They also create timely opportunities to engage customers on data modernization, AI-ready application development, secure infrastructure, and cloud-native operations. What was announced at Build 2026 Azure HorizonDB: A new standard for AI-native, enterprise PostgreSQL Azure HorizonDB enters public preview as a new PostgreSQL cloud database service engineered for performance, scale, and modern AI-powered application needs. For business leaders thinking about data strategy, this is significant. Organizations are under pressure to modernize legacy databases and support intelligent applications without sacrificing resilience, governance, or developer productivity. Azure HorizonDB is designed to address those priorities with a platform that can scale storage automatically for large enterprise workloads, scale compute across primary and replica nodes, and bring AI-native capabilities directly into the database layer. What stands out is that Azure HorizonDB gives enterprises a way to simplify architecture while also accelerating innovation. Features such as advanced filtered vector search, in-database AI model management, Microsoft Entra ID integration, and GitHub Copilot integration through the PostgreSQL extension for Visual Studio Code position it as more than a database modernization story. Developers can use Microsoft 365 Copilot with live database context to generate schema-aware SQL, explore database structures, analyze and rewrite queries, and build against HorizonDB-specific capabilities without leaving Visual Studio Code. It is designed to bring together open-source PostgreSQL, enterprise security, AI readiness, and a more integrated developer experience in a single managed service. For business leaders, that can create a faster path from data estate modernization to measurable business outcomes. In Microsoft internal testing environments, Azure HorizonDB performed three times faster than self-managed PostgreSQL. For partners, the announcement creates an opportunity to engage customers in PostgreSQL modernization, intelligent application architecture, migration planning, performance optimization, and AI-enabled development. Azure Linux: Open-source infrastructure at enterprise scale The Azure Linux public preview announcement is meaningful for leaders focused on cloud efficiency, security, and platform consistency. Linux is already foundational to modern digital infrastructure, and two-thirds of customer cores in Azure run Linux. Now Azure Linux provides a first-party Linux distribution, purpose-built for Azure, and is available for Azure virtual machines (VMs), VM scale sets, and container images. We also announced Azure Container Linux (ACL), a secure, immutable container host designed to help platform teams run Kubernetes workloads at scale on Azure Kubernetes Service (AKS). By bringing Azure Linux forward as a more visible first-party platform choice, Microsoft is giving organizations a cloud-native operating system designed for modern workloads, including virtual machines (VMs), containers, and AI infrastructure. This matters because infrastructure choices increasingly shape agility, security posture, and operating cost. Azure Linux reflects the Microsoft focus on secure-by-default design, consistent servicing, and tighter alignment between the operating system and the Azure platform. For enterprises, that can translate into simpler operations and a more predictable foundation for cloud-native applications. These announcements reinforce what the Microsoft partner ecosystem and customer usage have shown for years: open-source infrastructure is foundational to Microsoft cloud strategy, as more than 65% of customer cores in Azure run Linux. Continue reading blog hereWe Gave Ourselves 20 Minutes to Build an AI Agent for a Lumber Company. The Timer's Still on Screen.
Here's a confession: most "build with AI" webinars are 60 minutes of slides, 5 minutes of a polished demo someone rehearsed for a week, and a closing CTA. You leave inspired but not really sure what you saw. So we tried something different. We put a visible countdown timer on the screen and gave ourselves 20 minutes to do two things, live: Build an AI agent that solves a real business problem Deploy a working AI application to Azure No edits to hide the awkward parts. No "and here's one I prepared earlier." Just the timer, the screen, and a working app at the end. The on-demand recording is up now. Here's what's in it and why you should carve out 20 minutes for it this week. The setup: why lumber? 🏘️ We needed a real business problem, not a toy one. So for the demo, we role-play as the owner of Contoso Lumber — a regional lumber business with a very specific, very real headache: Should we sell our inventory now, or hold it longer? Sell too early, miss a better price. Hold too long, eat storage costs. Lumber prices fluctuate with global competition, macro shifts, even the weather. In the past, decisions like this came from morning meetings and gut instinct, or maybe the occasional ad-hoc spreadsheet that nobody could reuse a month later. It's the kind of decision that should have an analyst behind it — except most growing businesses can't afford to hire one full-time. So we build the AI agent that does. (Yes, lumber. We know. Stick with us — the boring industry is exactly the point. If it works here, it works for your business too.) What we actually build (in 20 minutes flat) The webinar walks through the entire flow, end to end: Part 1 — The agent. We open Microsoft Foundry at ai.azure.com, browse the model leaderboard (there are over 11,000 models to choose from — we compare a few on the cost-vs-quality chart), pick one, write a plain-English instruction for the agent, upload a CSV of historical lumber pricing, and ask it a real question: "If I cannot sell one of my products today unless I offer my clients a 35% discount, and knowing the historical pricing data, should I still sell it?" The agent runs a break-even analysis and comes back with a reasoned recommendation — hold for 3–6 months, here's the math on why, here's where storage costs start eating the upside. Then we add voice mode (now you can ask the agent for pricing recs from a coffee shop on your phone), and lock down guardrails to block jailbreaks, prompt injection, data leakage, and — because we're feeling fancy — profanity in responses. Part 2 — The app. With the agent done, we pivot to deploying a full AI chat application to Azure. From scratch. Using exactly five commands in Azure Cloud Shell: azd auth login git clone <repo> cd <folder> azd up azd down # (this one's for when you're done — kills everything to avoid surprise bills) That's it. The template handles the Container Apps setup, the architecture-aligned-to-Well-Architected-Framework stuff, all the boilerplate that usually eats half a sprint. By the end of the segment, there's a working AI chatbot running on a real Azure URL. We even pause the timer when we're just explaining things, so you know the 20-minute clock is honest about build time, not talk time. Why this format is more useful than another slide deck A few things this webinar shows that a written tutorial can't: The Foundry UI is super navigable. You watch someone do it. You see where the buttons are. You see what the leaderboard looks like when you're comparing GPT-5.3 Codex against Kimi K2.5 on a cost-to-quality chart. (Spoiler: Kimi wins this particular trio. Your mileage will vary depending on your workload.) The "no-stitching" claim is real. Models, data, agents, guardrails, deployment — all in one place. You don't need to leave Foundry to wire seven products together. The webinar makes that concrete by showing you the actual flow without cutting. Five commands really is five commands. This is the part people are most skeptical about until they see it. azd up does the work. The infrastructure provisioning, the container app, the AI service hookup — all of it. You can delete it just as fast. azd down tears everything back down. Useful when you're experimenting and don't want a $40 surprise on your Azure bill next month. What's on screen at the end By the 20-minute mark: A published AI agent named for the lumber business, with guardrails, voice mode enabled, ready to be called from Teams, Microsoft 365 Copilot, or any application via endpoint A separate AI chat application deployed to Azure Container Apps, with a live URL Logs, observability, the full Foundry control plane — all available out of the box And in the closing minutes, four very concrete next steps for what you do next if this sparked an idea for your own business — including Azure Accelerate (if you want Microsoft experts in the room with you), the partner network, and the Microsoft marketplace if you'd rather buy than build. Watch the recording The on-demand recording is available now. Block 20 minutes — that's literally all it takes — and ideally watch with your Azure portal open in another tab so you can follow along. If you're the kind of person who learns by doing, pause at the agent-building section and try it yourself in parallel. Foundry is free to explore; the agent we build in the webinar costs cents to run. → Watch the on-demand webinar A few things we'd love feedback on If you watch it, we'd genuinely love to know: Did the timer help or distract? (We thought it would feel gimmicky. It turned out to be the most-mentioned thing in early feedback.) What use case from your business would you want to see in the next one? We're picking the next demo problem from comments. Was the lumber thing weirdly compelling or were you just here for the Azure parts? Drop a comment, tag us, or grab a partner and try building your own version this week. The timer's reset. Your 20 minutes start whenever you press play. Want to go deeper than the webinar? Two companion reads: From Idea to Impact: How Growing Businesses Scale with Azure (five real customer stories with the full architectures) and AI Made Simple: 3 Practical Moves for Growing Businesses (the structured playbook for figuring out what to build first).
Productize, observe, version, and automate MCP servers in Azure API Management
Introduction As organizations move from AI-assisted applications to agentic workflows, MCP servers are becoming a critical integration layer between agents, tools, APIs, data sources, and enterprise systems. Azure API Management already helps teams bring MCP servers under enterprise governance. But as MCP adoption scales, platform teams need more than basic exposure. They need a way to package MCP servers for the right consumers, understand tool usage in detail, manage changes safely, and automate configuration across environments. These are familiar API management challenges — and the same patterns that organizations already use for APIs can now be applied more deeply to MCP servers. We are excited to announce new generally available capabilities for MCP server management in Azure API Management: Add MCP servers to products to package and govern MCP capabilities for specific consumers MCP tool observability to trace tool usage, logs, errors, and payload context MCP server versioning to run multiple versions side by side and manage change safely Management API and Bicep support to automate MCP server configuration as part of CI/CD workflows Together, these capabilities extend MCP server management in Azure API Management and help make MCP servers first-class managed resources — productized, observable, versionable, and automatable. Why MCP server management matters MCP gives agents a standard way to connect with tools and external capabilities. That standardization is powerful, but it also introduces a new operational surface for enterprises. Without a management layer, teams can quickly run into questions such as: Which MCP servers are approved for use? Who can access each server? How do we expose MCP servers to different developer or agent audiences? How do we monitor tool calls, latency, errors, and cost? How do we run preview and production versions side by side? How do we automate MCP server configuration across environments? These are not just developer experience questions. They are enterprise governance questions. With Azure API Management, MCP servers can now be managed using the same core patterns organizations already use for APIs: products, subscriptions, policies, observability, versioning, and automation. What’s new 1. Add MCP servers to products Azure API Management products are a proven way to package APIs for consumption. With this release, you can now add one or more MCP servers to APIM products as well. This makes it easier to expose MCP capabilities to specific consumers, teams, applications, or agent experiences using familiar product-based governance. For example, a platform team can create a product for internal agents that includes approved MCP servers such as: Customer profile lookup Order status retrieval Knowledge base search Ticket creation Workflow automation tools By adding MCP servers to products, teams can use familiar controls such as subscriptions, quotas, approval workflows, and access management to govern how MCP capabilities are consumed. Why it matters: MCP servers are no longer isolated endpoints. They can be bundled, governed, and delivered as secure, consumable products. 2. MCP tool observability As agents use MCP servers to discover and invoke tools, teams need more than basic traffic visibility. They need end-to-end trace context for each agent-to-tool interaction. With MCP observability in Azure API Management, teams can inspect key MCP-specific details, including: Operation context: whether the request was a tools/list or tools/call operation Session context: the MCP session ID through gen_ai.conversation.id Client context: MCP client name and version Protocol context: MCP protocol name and version Server context: MCP server name and version Access context: authentication type and API type Tool context: tool name and tool type for tool invocation traces Error context: error type and error message when a call fails Payload context: tool invocation arguments and results when payload logging is enabled This is especially important for agentic workflows, where a single user request may trigger multiple tool calls across different systems. With APIM, MCP traffic can be traced, inspected, and monitored using the same operational practices teams already use across their API estate. Why it matters: MCP servers are not just accessible through APIM — they are observable. Platform teams can trace tool calls, inspect errors, and understand MCP usage with the same operational discipline they expect from managed APIs. 3. Expose multiple MCP versions Enterprise teams need safe ways to evolve MCP servers over time. With MCP server versioning in Azure API Management, you can expose multiple versions of the same MCP server side by side. This allows teams to run a stable GA version while introducing a preview or next version for early adopters. For example: v1 can serve the majority of production traffic. v2 can be exposed to a subset of consumers for testing. Teams can monitor adoption, errors, latency, and behavior. Once the new version is validated, v2 can be promoted with confidence. This pattern is especially useful when MCP tools evolve, schemas change, new capabilities are added, or teams want to validate agent behavior before rolling changes out broadly. Why it matters: MCP servers can now follow a safer lifecycle model: preview, validate, route, promote, and retire. 4. Management API and Infrastructure as Code MCP server management also needs to work at enterprise scale. With Management API and Infrastructure as Code support, teams can provision and configure MCP servers programmatically through Azure API Management APIs and automation pipelines. This allows platform teams to define MCP server resources as part of repeatable deployment workflows using tools such as Bicep, Terraform, ARM, REST APIs, and CI/CD pipelines. Teams can automate configuration for: MCP server endpoints Runtime and transport settings Authentication configuration Metadata and ownership Versioning Product association Policies Environment promotion This is critical for organizations that need consistent MCP governance across development, test, staging, and production environments. Why it matters: MCP server management can now be automated, reviewed, deployed, and governed like the rest of your API platform. How these capabilities work together Individually, each capability solves an important operational need. Together, they create a complete management model for MCP servers in Azure API Management. A platform team can: Register or expose MCP servers through Azure API Management. Package them into products for specific consumers. Apply access controls, subscriptions, quotas, and policies. Observe tool-level usage, latency, errors, traces, and cost. Run multiple versions side by side. Promote changes safely. Automate deployment through APIs and Infrastructure as Code. This brings the full API management playbook to MCP. Instead of treating MCP servers as unmanaged agent extensions, organizations can operate them as governed enterprise resources. Example scenario Imagine a company building internal copilots for customer support, sales, and operations. Each copilot needs access to different tools: Customer lookup Order history Case management Knowledge search Refund workflows Escalation workflows With MCP and Azure API Management, the platform team can expose these capabilities as MCP servers and organize them into products. The customer support copilot can subscribe to the support product. The sales copilot can subscribe to the sales product. Early adopters can be routed to a preview version of a tool. Operations teams can monitor usage, errors, latency, traces, and cost. Platform teams can automate the entire setup across environments. The result is a more governed and scalable way to bring MCP-based tools into enterprise agent workflows. Getting started To get started with MCP server management in Azure API Management: Create or identify an MCP server you want to expose through Azure API Management. Add the MCP server as a managed resource in APIM. Add the MCP server to an APIM product. Configure access, subscriptions, quotas, and approval workflows. Enable observability to monitor tool-level usage and traces. Use versioning to manage preview and production versions. Use the Management API or Infrastructure as Code to automate configuration. Conclusion MCP is quickly becoming an important standard for connecting agents to tools and enterprise capabilities. But for MCP to succeed in production, organizations need more than connectivity. They need governance, lifecycle management, observability, and automation. With these new MCP server management capabilities in Azure API Management, platform teams can manage MCP servers using the same trusted patterns they already use for APIs. MCP servers are now first-class APIM resources — productized, observable, versionable, and automatable. We are excited to see how customers use these capabilities to build the next generation of governed, enterprise-ready agentic applications.Unlocking the Human Telemetry Layer for Safer Industrial Operations
What if we could track human health & safety conditions as precisely as we do with machines, and take immediate actions to protect our greatest asset, our people? Many industrial organizations still lack visibility into real-time human conditions, even as worker safety and operational risk remain major investment priorities. One of the most important operational signals has largely remained outside the industrial data estate: the human telemetry. VOORMI and Microsoft have joined forces to fill this gap in understanding real human conditions. Through the Mij™ platform, VOORMI brings human telemetry into Azure IoT, enabling enterprises to integrate worker conditions such as heat stress and fatigue into the same operational architecture already used for machines and industrial systems. VOORMI, SWNR’s performance apparel brand, is among the first to bring this technology into garments designed for real industrial field conditions. This integration brings their proprietary wearable technology directly into high-impact worker safety and field operations scenarios. The partnership helps establish a new telemetry layer for industrial operations, allowing human, machine, and environmental signals to converge and drive safer operations, real-time awareness, and adaptive AI workflows. Bringing the Human Signal into Industrial AI with Azure Industrial organizations increasingly recognize that many safety, productivity, and operational challenges occur at the intersection of people and machines. Workers operate in high-heat environments, hazardous conditions, remote sites, and physically demanding field scenarios where situational awareness matters in real time. Historically, worker telemetry has remained fragmented across proprietary wearable platforms and disconnected safety systems, creating governance and operational challenges for enterprise IT and OT teams. Mij™ is designed differently, integrating directly into customer-controlled Azure environments through Azure IoT Operations running at the edge or Azure IoT Hub in the cloud rather than introducing another isolated platform. Running intelligence at the edge enables virtual safety agents and operational workflows to execute closer to the worker, supporting low-latency responses, local interaction with OT systems, and operational resilience even in disconnected or bandwidth-constrained environments. This gives enterprises flexibility to support real-time worker safety responses at the edge while also enabling long-term analytics, reporting, and operational intelligence through Microsoft Fabric. Telemetry from garment-integrated sensors flows through edge gateways into Azure services including Azure IoT Operations, Azure Data Explorer, Azure Managed Grafana, and Microsoft Fabric. The result is a unified operational environment where worker telemetry can live beside machine, site, and environmental data under the customer’s existing identity, security, governance, and analytics model. The vision is simple and transformative: make human telemetry a trusted, first-class industrial data source. Azure Digital Operations as the Intelligence Layer The reference architecture demonstrates how Azure IoT Operations can serve as a scalable operational intelligence layer for worker safety and connected operations scenarios across manufacturing, energy, and field environments. Mij™-enabled garments broadcast Bluetooth Low Energy (BLE) telemetry that can be processed locally through edge gateways and routed into Azure IoT Operations using MQTT and dataflows. Data is then operationalized through Azure Data Explorer and visualized using Azure Managed Grafana dashboards for field operations, worker safety, fleet health, gateway monitoring, and operational readiness scenarios. Telemetry can also be made available to Foundry Local-hosted GenAI agents to support real-time, context aware safety guidance, such as prompting workers operating in high-heat conditions to hydrate or seek cooler environments. While Mij™-enabled garments are the initial implementation, the edge device-to-cloud architecture creates a broader onboarding point for additional wearable, sensor, and field telemetry scenarios over time. This allows enterprises to bring more human and operational signals into a unified Azure-native operational environment. The architecture also supports flexible ingestion patterns for environments where dedicated edge gateways are not practical. Using Microsoft Entra External ID, Azure Container Apps, and Azure IoT Hub, telemetry can securely flow into Azure services without exposing operational infrastructure credentials to client devices. This pattern aligns with the broader Azure adaptive cloud approach: enabling customers to run distributed edge-native services on Arc-enabled Kubernetes infrastructure while maintaining centralized security, governance, and analytics capabilities across the enterprise. Depending on customer architecture preferences, telemetry can be processed through Azure IoT Operations at the edge or ingested directly through Azure IoT Hub for cloud-first analytics and downstream processing in services such as Microsoft Fabric. Edge processing also enables real-time sensor fusion across worker telemetry, ambient environmental conditions, machine parameters, and site-level operational signals, supporting faster safety interventions and more context-aware operational decisions. This gives enterprises flexibility in how they balance edge processing, operational responsiveness, governance and privacy requirements. Enabling the Next Generation of Industrial Workflows The long-term opportunity extends well beyond visualization dashboards. As worker telemetry becomes part of the operational fabric, enterprises can begin building more adaptive and intelligent workflows across worker safety, field readiness, incident response, compliance, environmental monitoring, and industrial AI systems. Human telemetry can provide critical real-time context that complements machine and environmental signals enabling more responsive operations and eventually more autonomous decision-support experiences. By bringing human telemetry into enterprise AI and analytics workflows, organizations can build more adaptive operational systems that improve worker safety, situational awareness, and real-time decision making at scale. This partnership reflects a broader industry shift: industrial transformation is no longer only about connected machines. It is about connected operations where people, equipment, environments, and AI systems participate in a shared operational intelligence layer. With SWNR’s Mij™platform and Azure IoT Operations, Microsoft and VOORMI are helping unlock that future. Learn more: Mij™ product page: https://swnrtechnologies.com/pages/mij Learn more about Azure IoT Operations: Documentation & Getting Started See what’s new with Azure IoT Hub: Preview Documentation To get started with a pilot, contact: pilots@swnrtechnologies.comIntroducing Durable Functions in PostgreSQL
By Abe Omorogbe, Senior PM | Pino De Candia, Principal Software Engineer | TJ Green, Principal Software Engineer Postgres will happily store your data, run your queries, and scale with you for years. But the moment you need to do more with that data, such as running multi-step transformation, scheduling nightly rollups, generating embeddings or waiting on an approval, you hit a wall. Postgres has no built-in way to run long-lived, fault-tolerant work. That's why we built pg_durable, a new open-source PostgreSQL extension that brings durable execution directly into the database. With pg_durable, Postgres doesn’t just store your data, it runs long-lived, fault-tolerant workflows on it, with built-in retries, parallelism, scheduling, and recovery. Instead of stitching together PL/pgSQL functions or building external orchestration systems, you can now define and run resilient workflows entirely in your database, backed by Postgres' durability and high availability. And on Azure HorizonDB, pg_durable also powers AI pipelines, enabling production-ready data and AI workflows, end-to-end, right inside the database. In this post, we'll cover: The hidden trap: blocking background work What pg_durable is and the DSL that drives it How this engine powers AI pipelines on HorizonDB Sample patterns worth exploring Getting started on HorizonDB, on your laptop, and in VS Code 🚀 Want to try it out? pg_durable ships in Azure HorizonDB, Microsoft's new PostgreSQL cloud service. The HorizonDB Preview is the fastest way to try pg_durable and AI pipelines together. Get started in HorizonDB → pg_durable visualization The hidden trap: blocking background work Most Postgres teams eventually reach a point where they need to run critical tasks on their data: transformations, nightly aggregations, database maintenance workflows, embedding jobs, or multi-step business processes. So, they do the natural thing and try to keep that work inside Postgres. They end up on a journey of increasing complexity and maintenance burden. First, just run the task as a function in your database You cram the whole workflow into one PL/pgSQL function: loop, transform, call APIs, write results, return. It looks simple until you have to run it in production. One connection stays tied up the whole time. Everything runs inside one big transaction, with long locks and no visibility into partial progress. If the connection drops or the database restarts, the whole run is gone. No per-step retries. No parallelism. No scheduling. No clean way to pause for human input. When it fails, you move it outside You push the workflow into an external service: a job queue, polling workers, state tables, step coordination, retry logic, crash-recovery sweeps, and cleanup jobs. What started as a few background tasks turns into a full distributed system. Before you’ve even touched the business logic, you’re building and operating infrastructure just to coordinate work that’s still fundamentally tied to your data. Both paths are workarounds for the same missing primitive: durable, asynchronous background work that lives where your data lives. That's the gap pg_durable fills. What pg_durable actually is pg_durable is a Postgres extension that consists of a DSL (Domain specific language) and the duroxide runtime hosted in a Postgres background worker. You describe a workflow as a small SQL expression, call df.start(...), and get an instance ID back immediately. The work runs off to the side in a background worker, so it never blocks your connection or transaction, and you can check progress later with df.status() and df.result(). The execution state lives in Postgres, which means it benefits from the database’s durability, HA, backups, and recovery. Additionally, the workflow definition does not have to live in the database: your application can send it to df.start(...) over a regular Postgres connection. 2: pg_durable orchestration of worker and schema Because execution is asynchronous, pg_durable automatically breaks a workflow into discrete steps. Each step runs in its own session and transaction, commits its progress, and hands off to the next instead of keeping one giant transaction open. Steps are checkpointed in Postgres and recovered by deterministic replay, so workflows survive crashes, restarts, and failovers and resume where they left off. If a step fails, only that step retries. The whole thing is expressed through a tiny DSL of composable operators: Operator Meaning ~> Sequential. run this, then that & Parallel. fan out, wait for all | Race. fan out, take the first to finish ?> / !> Conditional. if / else @> Loop. repeat durably, survive restarts |=> Capture a step's result into a variable (reuse with $) Advanced Functions df.if() Conditional branch df.loop() Repeat statements df.join() Execute in parallel, wait for all df.http() To call an allowlisted endpoint df.wait_for_schedule() For cron-style timing df.wait_for_signal() Pause for an external event Read more about all operators and functions in pg_durable Without pg_durable vs. with pg_durable The hand-rolled version of "run three aggregations in parallel, then refresh a dashboard with retries and crash recovery" usually means 300+ lines of queue tables, polling workers, state-machine rows, per-step retry logic, crash-recovery sweeps, and cleanup jobs. Plus, the runbook to operate it. The pg_durable version: SELECT df.start( 'SELECT count(*) FROM users' & 'SELECT count(*) FROM orders' & 'SELECT sum(amount) FROM orders' ~> 'REFRESH MATERIALIZED VIEW metrics', 'refresh-dashboard' ); You write the SQL. pg_durable owns the queue, the state, the coordination, the retries, and the crash recovery. Two ways to use pg_durable 1: Use pg_durable directly (works on Azure HorizonDB or any Postgres 17) Enable it and start orchestrating: CREATE EXTENSION pg_durable; SELECT df.start($$ SELECT 'Hello, durable world!' AS message $$); -- returns an instance ID immediately; the worker runs it asynchronously From there you compose: sequential pipelines, conditional branches, races for timeout-or-result, variable passing between steps, human-in-the-loop approvals, scheduled maintenance all in SQL, close to the data, with no new infrastructure. This is the "just use Postgres" answer to a problem teams usually solve by leaving Postgres. Because it's open source under the permissive PostgreSQL License, you can clone the repo and run it on your laptop, your server, or any cloud. 2: AI pipelines (HorizonDB capability) On HorizonDB, pg_durable becomes the foundation for something even more approachable: a managed, declarative AI pipeline surface in the azure_ai extension. pg_durable gives you the durable execution engine, while the ai.* API gives you an AI-shaped model of sources, steps, sinks, and triggers that compile into a durable graph. Traditional app-tier embedding pipelines fail in predictable ways: a transient API error mid-batch with no shared checkpoint, a worker that crashes after writing chunks but before marking the parent row processed, no clean way to re-embed just the rows that changed. Move that logic into HorizonDB and the source, the steps, the sink, and the run history are all SQL, protected by the same transactions, backups, and PITR (point-in-time recovery) your data already has. A complete chunk → embed AI pipeline is one definition: SELECT ai.create_pipeline( name => 'ai_pipeline', source => ai.table_source(table_name => 'documents_ai_pipeline'), steps => ARRAY[ ai.chunk(input => 'content'), ai.embed(model => 'default-embedding', input => 'chunk_text', dimensions => 1536) ], trigger => 'on_change', sink => ai.table_sink('documents_ai_pipeline_output') ); SELECT ai.run('ai_pipeline'); Each AI step becomes a durable node, so if ai.embed() fails, ai.chunk() doesn’t run again. And with trigger => 'on_change', the pipeline runs automatically as rows change, embedding only what’s new. Add a DiskANN index on the resulting table, and you have production-ready vector search end to end, entirely inside the database. Where pg_durable fits and where it doesn't If you've used external orchestrators such as Temporal or Airflow, your first reaction is probably: why would I put control flow in my database? Fair question. pg_durable isn't trying to be a universal orchestrator. Reach for pg_durable when the workflow is tightly coupled to Postgres state. The rows it reads and writes live in the same database, it benefits from the database's own durability, backups, and PITR, and you'd rather not stand up a separate system to coordinate work that never leaves the data tier. Think: embedding pipelines, ETL jobs, scheduled maintenance, and queue-style background jobs. Reach for a dedicated orchestrator when the workflow's center of gravity is outside Postgres, fanning across heterogeneous services, or running arbitrary application logic that does not map cleanly to SQL steps, branching, loops, or HTTP calls. Get started On Azure HorizonDB CREATE EXTENSION IF NOT EXISTS pg_durable; -- Execute a simple SQL query as a durable function SELECT df.start($$ SELECT 'Hello, durable world!' AS message $$); -- Returns: a1b2c3d4 (8-character instance ID) -- Get result of a specific instance SELECT df.result(<ID>); That's it: submit, walk away, inspect. Read the documentation for more details. In VS Code, with the PostgreSQL extension A dense one-liner of ~>, &, and |=> is precise once it clicks, but the learning curve is real so flatten it with tooling. Install the PostgreSQL extension for VS Code from the Marketplace: Connect to HorizonDB or your local Postgres directly from the extension Let Copilot write the SQL. The pg-durable-sql skill turns a plain-English description ("every night, archive orders older than 90 days") into correct pg_durable syntax. Run it and watch it. The extension renders pg_durable workflows and azure_ai pipelines as live graphs, definition and each run, so you can see every step, its timing, and exactly where a failure happened. Authoring, execution, run visualization, and inspection in one window and the same tooling works against any Postgres, not just HorizonDB. On your laptop Prefer to run it yourself? Clone microsoft/pg_durable, use the Codespace prebuild or VS Code Dev Container, and add the extension on any Postgres 17. Sample patterns worth exploring The scenario guide has a full catalog of scenarios; however, these are the three I would start with. ETL Pipeline: a multi-step data transformation where each step must be completed before the next begins. Failures should stop the pipeline. SELECT df.start( 'DELETE FROM target WHERE loaded_at < now() - interval ''7 days''' -- Step 1: Cleanup old ~> 'UPDATE staging SET processed_at = now() WHERE processed_at IS NULL' -- Step 2: Mark staging ~> 'INSERT INTO target (data, source_id) SELECT data, source_id FROM staging WHERE processed_at IS NOT NULL', -- Step 3: Load 'etl-pipeline' -- Label for easy identification ); If the database restarts mid-backfill, it picks up from the last checkpointed batch, not row zero. See full example Scheduled Data Sync: poll an external API or run a job on a schedule (hourly, daily, every 30 minutes). The job should run forever and survive restarts. (See full example): -- Scheduled sync: fetch data every 30 minutes (runs forever) SELECT df.start( @> ( -- @> creates an eternal loop -- Fetch from external API (df.http( 'https://httpbingo.org/json', 'GET' ) |=> 'response') -- Store the response ~> 'INSERT INTO external_data_sync (data) VALUES ($response::jsonb)' -- Wait for next scheduled run ~> df.wait_for_schedule('*/30 * * * *') -- Cron: every 30 minutes ), 'scheduled-data-sync' ); Human-in-the-loop approval: auto-apply routine changes, pause the risky ones until a person signals approval (See full example): SELECT df.start( 'SELECT amount > 10000 AS needs_review FROM invoices WHERE id = 42' |=> 'risky' ?> ( df.wait_for_signal('invoice-42') ~> 'UPDATE invoices SET status = ''paid'' WHERE id = 42' ) !> 'UPDATE invoices SET status = ''paid'' WHERE id = 42', 'invoice-approval' ); The workflow simply waits minutes or days until a reviewer releases it with the matching signal, then resumes. The community is already running with it pg_durable launched as open source and the community is already kicking the tires. The project was a top article on Hacker News on launch day and 1.7K stars on GitHub within its first few days of initial launch. Also Franck Pachot (PostgreSQL community veteran) published an independent walkthrough, Getting Started with pg_durable: durable workflows inside PostgreSQL within days of release. The repo is actively developed, and the maintainers are reading every issue and PR. If you want improvements in our DSL ergonomics, say so. If you want an operator that doesn't exist yet, open an issue. If you've got a scenario we haven't covered, send a PR. The syntax, the docs, and the rough edges all get better when people who run Postgres in production push back. So, clone it, and build something real. If you find rough edges, open an issue or send a PR at microsoft/pg_durable. We think you'll be surprised by how much it can take. Learn more pg_durable on GitHub Durable Functions on HorizonDB AI pipelines on HorizonDBPatner Case Study | Pax8
From their early days of selling Microsoft licenses to becoming a leading cloud commerce marketplace, Pax8 has always centered their business on the Microsoft partner experience. And their future-focused, partner first approach is redefining what it means to be a modern Cloud Solution Provider (CSP). For Pax8, the next evolution of that role is the Managed Intelligence Provider (MIP)—a partner that moves beyond managing systems to orchestrating intelligence and delivering measurable business outcomes with AI. While Pax8 delivers critical value through the streamlined billing and provisioning of Microsoft licenses, they also know that real business transformation doesn’t happen just because you’ve purchased software—it happens because of everything that comes after the sale. That’s why they invested heavily in enablement, education, and professional services. And that’s why today, they’re successfully cultivating an ecosystem that drives meaningful, scalable growth for partners and customers alike. Grounded in their experience as a member of the Microsoft AI Cloud Partner Program, Pax8’s approach has changed the trajectories of organizations like Sourcepass, a US-based managed services provider (MSP). Together, Sourcepass and Pax8 are making enterprise-grade AI accessible to small and medium-sized businesses (SMBs)—but they’re going beyond traditional deployments. They’re delivering functional, agentic solutions designed to help customers realize value and support recurring service opportunities. Navigating AI adoption in the SMB reality SMBs are under constant pressure to do more with less, and the business outcomes they need the most—efficiency, strong security, scalable growth—are the exact outcomes AI can deliver. But with limited budgets, it’s hard for SMBs to justify the spend without clear proof of value—no matter how interested they might be. “SMBs hear about AI and want to be part of the wave,” said Elliott Eliason, Productivity Solution Consultant at Pax8, “but they don’t want to spend thousands without getting results back.” And once they’ve made an investment in AI, they likely lack the internal resources to operationalize it. That includes preparing data governance, validating security controls, and teaching teams new workflows. For a partner like Sourcepass, who primarily serves SMBs, this means their customers need trusted, strategic advisors who can guide them in turning AI’s potential into tangible outcomes. Continue reading here Explore all case studies or submit your own Subscribe to case studies tag to follow all new case study posts. Don't forget to follow this blog to receive email notifications of new stories!
From AI Suggestions to Autonomous CRM Actions in Dynamics 365
Modern CRM AI solutions often stop at case summarization—but real transformation requires more. This blog introduces a CRM Copilot Agent Accelerator built on Microsoft Power Platform, designed to evolve AI from simple insights to predictive intelligence and ultimately to autonomous actions. By combining Dynamics 365, Dataverse, Power Automate, and AI Builder, and extending capabilities through modular add-on packs, this approach enables organizations to reduce manual effort, improve decision-making, and scale service operations efficiently—without additional Copilot licensing.
