Azure App Service Managed Instances: What IT/Ops Teams Need to Know
Azure App Service has long been one of the most reliable ways to run web apps on Azure, giving teams a fully managed platform with built‑in scaling, deployment integration, and enterprise‑grade security. But for organizations that need more control, expanded flexibility, or the ability to run apps that have additional dependencies, the new Azure App Service Managed Instance (preview) brings a powerful new option. Vinicius Apolinario recently sat down with Andrew Westgarth, Product Manager for Azure App Service to talk through what Managed Instances are, why they matter, and how IT/Ops teams can take advantage of the new capabilities. What Managed Instances Bring to the Table Managed Instances (MI) deliver the App Service experience you know with added flexibility for additional scenarios. You get the same PaaS benefits—patching, scaling, deployment workflows—but with the control typically associated with IaaS. Some of the highlights we discussed: App Service and App Service Managed Instance — What are the main differences and what scenarios MI is focusing on. Consistent App Service experience — Same deployment model, same runtime options, same operational model. App service experience for different audiences — How IT/Ops teams can leverage MI and what does it mean for development teams. Features IT/Ops Teams Will Appreciate Beyond the core architecture, MI introduces capabilities that make day‑to‑day operations easier: Configuration (Install) Script — A new way to customize the underlying environment with scripts that run during provisioning. This is especially useful for installing dependencies, configuring app and OS settings, installing fonts, or preparing the environment for the workload. RDP Access for Troubleshooting — A long‑requested feature that gives operators a secure way to RDP into the instance for deep troubleshooting. Perfect for diagnosing issues that require OS‑level visibility. Learn more about Azure App Service Managed Instance (preview): Documentation: https://aka.ms/AppService/ManagedInstance Hands On Lab: https://aka.ms/managedinstanceonappservicelab Blog: https://aka.ms/managedinstanceonappservice Ignite session: https://ignite.microsoft.com/en-US/sessions/BRK102Azure AI Connect - March 2 to March 6 2026
The Future of AI is Connected. The Future is on Azure. Join us for a 5-day virtual event dedicated to mastering the Microsoft Azure AI platform. Azure AI Connect isn't just another virtual conference. It's a 5-day deep-dive immersion into the *connective tissue* of artificial intelligence on the cloud. We're bringing together developers, data scientists, and enterprise leaders to explore the full spectrum of Azure AI services—from Cognitive Services and Machine Learning to the latest breakthroughs in Generative AI. Explore the Ecosystem: Understand how services work *together* to create powerful, end-to-end solutions. Learn from Experts: Get direct insights from Microsoft MVPs, product teams, and industry pioneers. Gain Practical Skills: Move beyond theory with code-driven sessions, practical workshops, and live Q&As. Connect with Peers: Network with a global community in our virtual lounge. Event DetailsHow to Build a Microsoft Marketplace Channel Practice: A Partner Guide to Scalable Growth
For Microsoft partners looking to make Marketplace a repeatable, scalable growth channel, this article outlines what it really takes to succeed. It breaks down how top partners structure their teams, align ownership across sales, alliances, and operations, and activate Microsoft Marketplace as a core part of their co-sell and go-to-market strategy. With practical frameworks, best practices, and real-world guidance, this resource helps partners drive pipeline, accelerate deal velocity, and grow alongside Microsoft through Marketplace. Read the full article: How to build a Microsoft Marketplace channel practice
Cloud Kerberos Trust with 1 AD and 6 M365 Tenants?
Hi, we would like to enable Cloud Kerberos Trust on hybrid joined devices ( via Entra connect sync) In our local AD wie have 6 OUs and users and devices from each OU have a seperate SCP to differnt M365 Tenants. I found this Article to configure the Cloud Kerberos Trust . Set-AzureADKerberosServer 1 2 The Set-AzureADKerberosServer PowerShell cmdlet is used to configure a Microsoft Entra (formerly Azure AD) Kerberos server object. This enables seamless Single Sign-On (SSO) for on-premises resources using modern authentication methods like FIDO2 security keys or Windows Hello for Business. Steps to Configure the Kerberos Server 1. Prerequisites Ensure your environment meets the following: Devices must run Windows 10 version 2004 or later. Domain Controllers must run Windows Server 2016 or later. Install the AzureADHybridAuthenticationManagement module: [Net.ServicePointManager]::SecurityProtocol = [Net.ServicePointManager]::SecurityProtocol -bor [Net.SecurityProtocolType]::Tls12 Install-Module -Name AzureADHybridAuthenticationManagement -AllowClobber 2. Create the Kerberos Server Object Run the following PowerShell commands to create and publish the Kerberos server object: Prompt for All Credentials: $domain = $env:USERDNSDOMAIN $cloudCred = Get-Credential -Message 'Enter Azure AD Hybrid Identity Administrator credentials' $domainCred = Get-Credential -Message 'Enter Domain Admin credentials' Set-AzureADKerberosServer -Domain $domain -CloudCredential $cloudCred -DomainCredential $domainCred As I understand the process, a object is created in local AD when running Set-AzureADKerberosServer What happens, if I run the command multiple times, for each OU/Tenant. Does this ovveride the object, or does it create a new objects?Implementing / Migrating the BizTalk Server Aggregator Pattern to Azure Logic Apps Standard
While the article focuses on the migration path from BizTalk Server, the template is equally suited for new (greenfield) implementations any team looking to implement the Aggregator pattern natively in Azure Logic Apps can deploy it directly from the Azure portal without prior BizTalk experience. The template source code is open source and available in the Azure LogicAppsTemplates GitHub repository. For full details on the original BizTalk implementation, see the BizTalk Server Aggregator SDK sample. Why is it important? BizTalk Server End of life has been confirmed and if you have not started your migration to Logic Apps, you should start soon. This is one of many articles in BizTalk Migration. More information can be found here: https://aka.ms/biztalkeolblog. The migration at a glance: BizTalk orchestration vs. Logic Apps workflow The BizTalk SDK implements the pattern through an orchestration (Aggregate.odx) that uses correlation sets, receive shapes, loop constructs, and send pipelines. The Logic Apps Standard template replicates the same logic using a stateful workflow with Azure Service Bus and CorrelationId-based grouping. The BizTalk solution includes: Component Purpose Aggregate.odx Main orchestration that collects correlated messages and executes the send pipeline FFReceivePipeline.btp Receive pipeline with flat file disassembler Invoice.xsd Document schema for invoice messages InvoiceEnvelope.xsd Envelope schema for output interchange PropertySchema.xsd Property schema with promoted properties for correlation XMLAggregatingPipeline.btp Send pipeline to assemble collected messages into XML interchange The Azure Logic Apps Standard implementation The Logic Apps Standard workflow replicates the same Aggregator pattern using a stateful workflow with Azure Service Bus as the message source and CorrelationId-based grouping. The template is publicly available in the Azure portal templates gallery. Figure 2: The “Aggregate messages from Azure Service Bus by CorrelationId” template in the Azure portal templates gallery, published by Microsoft. Receives messages from Service Bus in batches, groups them by CorrelationId, decodes flat files, and responses with the aggregated result via HTTP. Side-by-side comparison: BizTalk Server vs. Azure Logic Apps Understanding how each component maps between platforms is essential for a smooth migration: Concept BizTalk Server (Aggregate.odx) Azure Logic Apps Standard Messaging infrastructure MessageBox database (SQL Server) Azure Service Bus (cloud-native PaaS) Message source Receive Port / Receive Location Service Bus trigger (peekLockQueueMessagesV2) Message decoding Receive Pipeline (Flat File Disassembler) Decode_Flat_File_Invoice action (FlatFileDecoding) Correlation mechanism Correlation Sets on promoted properties (DestinationPartnerURI) CorrelationId from Service Bus message properties Message accumulation Loop shape + Message Assignment shapes ForEach loop + CorrelationGroups dictionary variable Completion condition Loop exit (10 messages or 1-minute timeout) Batch-based: processes all messages in current batch Aggregated message construction Construct Message shape + XMLAggregatingPipeline Build_Aggregated_Messages ForEach + Compose actions Result delivery Send Port (file, HTTP, or other adapter) HTTP Response or any other regarding business need Error handling Exception Handler shapes + SuspendMessage.odx Scope + error handler actions Schema support BizTalk Flat File Schemas (XSD) Same XSD schemas in Artifacts/Schemas folder State management Orchestration dehydration/rehydration Stateful workflow with run history Key architectural differences Aspect BizTalk Server Azure Logic Apps Standard Processing model Convoy pattern (long-running, event-driven) Batch-based (processes N messages per trigger) Scalability BizTalk Host instances (manual scaling) Elastic scale (Azure App Service Plan) Retry logic Adapter-level retries Built-in HTTP retry policy (3 attempts, 10s interval) Architecture Monolithic orchestration Decoupled: aggregation + downstream processing Monitoring BizTalk Admin Console / HAT Azure portal run history + Azure Monitor Schema reuse BizTalk project schemas Direct XSD reuse in Artifacts/Schemas Deployment MSI / BizTalk deployment ARM templates, Azure DevOps, GitHub Actions How the workflow works 1. Trigger: Receive messages from Service Bus The workflow uses the built-in Service Bus trigger to retrieve messages in batches from a non-session queue. This is analogous to BizTalk's Receive Location polling the message source. 2. Process and correlate: Group messages by CorrelationId Each message is processed sequentially (like BizTalk's ordered delivery). The workflow: Extracts the CorrelationId from Service Bus message properties (equivalent to BizTalk's promoted property used in the Correlation Set) Decodes flat file content with zero refactoring using the XSD schema (equivalent to BizTalk's Flat File Disassembler pipeline component) Groups messages into a dictionary keyed by CorrelationId (equivalent to BizTalk's loop + message assignment pattern) 3. Build aggregated output Once all messages in the batch are processed, the workflow constructs a result object for each correlation group containing the CorrelationId, message count and the array of decoded messages. 4. Deliver results The aggregated output is sent to a target workflow via HTTP POST, following a decoupled architecture pattern. This is analogous to BizTalk's Send Port delivering the result to the destination system. You can substitute this action for another endpoint as needed. This, will depend on your business case. Azure Service Bus: The cloud-native replacement for BizTalk’s MessageBox In BizTalk Server, the MessageBox database is the central hub for all message routing, subscription-based delivery, and correlation. It’s the engine that enables patterns like the Aggregator — messages are published to the MessageBox, and the orchestration subscribes to them based on promoted properties and correlation sets. In Azure Logic Apps Standard, there is no MessaeBox equivalent. Instead, Azure Service Bus serves as the cloud-native messaging backbone. Service Bus provides the same publish/subscribe semantics, message correlation (via the built-in CorrelationId property), peek-lock delivery, and reliable queuing — but as a fully managed, elastically scalable PaaS service with no infrastructure to maintain. This is a fundamental shift in architecture: you move from a centralized SQL Server-based message broker (MessageBox) to a distributed, cloud-native messaging service (Service Bus) that scales independently and integrates natively with Logic Apps through the built-in Service Bus connector. Important: Service Bus is not available on-premises. However, RabbitMQ is available to cover these needs, on-premises. RabbitMQ offers a fantastic alternative for customers looking to replicate BizTalk message routing, subscription-based delivery, and correlation. Decode Flat File Invoice: Reuse your BizTalk schemas with zero refactoring One of the biggest concerns during any BizTalk migration is: “What happens to our flat file schemas and message formats?” The workflow template includes a Decode Flat File action (type FlatFileDecoding) that converts positional or delimited flat file content into XML — exactly like BizTalk’s Flat File Disassembler pipeline component. The key advantage: your original BizTalk XSD flat file schemas work as-is. Upload them to the Logic Apps Artifacts/Schemas folder and reference them by name in the workflow — no modifications, no refactoring. This means: Your existing message formats don’t change — upstream and downstream systems continue sending and receiving the same flat file messages Your BizTalk schemas are directly reusable — the same Invoice.xsd from your BizTalk project works seamlessly with the FlatFileDecoding action Migration effort is significantly reduced — no need to redesign schemas, re-validate message structures, or update trading partner agreements Time-to-production is faster — focus on workflow logic and connectivity instead of rewriting message definitions Notice that, if you need to process XML data, as your data might arrive in XML format, use the XML Operations: Validate, Transform, Parse, and Compose XML with schema. You can find more information at Compose XML using Schemas in Standard Workflows - Azure Logic Apps | Microsoft Learn. The message with correlation Id Each message in the Service Bus queue is a flat file invoice the same positional/delimited text format used in the BizTalk SDK sample. Here's an example: INVOICE12345 DestinationPartnerURI:http://www.contoso.com?ID=1E1B9646-48CF-41dd-A0C0-1014B1CE5064 BILLTO,US,John Connor,123 Cedar Street,Mill Valley,CA,90952 101-TT Plastic flowers 10 4.99 Fragile handle with care 202-RR Fertilizer 1 10.99 Lawn fertilizer 453-XS Weed killer 1 5.99 Lawn weed killer The message structure combines positional and delimited fields: Line 1: Invoice identifier (fixed-length record) Line 2: Destination partner URI — in BizTalk, this value is promoted as a context property and used in the Correlation Set to group related messages Line 3: Bill-to header (comma-delimited: country, name, address, city, state, ZIP) Line 4: Line items (positional fields: item code, description, quantity, unit price, notes) Why CorrelationId is essential In BizTalk Server, the orchestration promotes `DestinationPartnerURI` from the message body into a context property and uses it as the Correlation Set to match related messages. This requires a Property Schema, promoted properties, and pipeline configuration. In Azure Logic Apps Standard, correlation is decoupled from the message body. The `CorrelationId` is a native Azure Service Bus message property with a first-class header set by the message producer when sending to the queue. This means: No schema changes needed: the flat file content stays exactly the same No property promotion: Service Bus provides the correlation identifier out of the box Simpler architecture: the workflow reads `CorrelationId` directly from the message metadata, not from the payload Producer flexibility any system sending to Service Bus can set the `CorrelationId` header using standard SDK methods, without modifying the message body This is why the Aggregator pattern maps so naturally to Service Bus: the correlation mechanism that BizTalk implements through promoted properties and correlation sets is already built into the messaging infrastructure. Step-by-step guide: Deploy the template from the Azure portal The “Aggregate messages from Azure Service Bus by CorrelationId” template is publicly available in the Azure Logic Apps Standard templates gallery. Follow these steps to deploy it: Prerequisites Before you begin, make sure you have: An Azure subscription. If you don’t have one, sign up for a free Azure account . An Azure Logic Apps Standard resource deployed in your subscription. If you need to create one, see Create an example Standard logic app workflow . An Azure Service Bus namespace with a non-session queue configured. A flat file XSD schema (for example, Invoice.xsd) ready to upload to the logic app’s Artifacts/Schemas folder. A target workflow with an HTTP trigger to receive the aggregated results (optional, can be created after deployment). Step 1: Open the templates gallery Sign in to the Azure portal. Navigate to your Standard logic app resource. On the logic app sidebar menu, select Workflows. On the logic app sidebar menu, select Workflows. On the Workflows page, select + Create to create a new workflow. In the “Create a new workflow” pane, select Use Template to open the templates gallery and select Create button. Step 2: Find the Aggregator template In the templates gallery, use the search box and type “Aggregate” or “Aggregator”. Optionally, filter by: o Connectors: Select Azure Service Bus o Categories: All Locate the template named “Aggregate messages from Azure Service Bus by CorrelationId”. o The template card shows the labels Workflow and Event as the solution type and trigger type. o The template is published by Microsoft. Step 3: Review the template details Select the template card to open the template overview pane. On the Summary tab, review: o Connections included in this template: Azure Service Bus (in-app connector) o Prerequisites: Requirements for Azure Service Bus, flat file schema, and connection configuration o Details: Description of the Aggregator enterprise integration pattern implementation Source code: Link to the GitHub repository Select the Workflow tab to preview the workflow design that the template creates and when you are ready select Use this template. Step 4: Provide workflow information In the Create a new workflow from template pane, on the Basics tab: o Workflow name: Enter a name for your workflow, for example, wf-aggregator-invoices o State type: Select Stateful (recommended for aggregation scenarios that require run history and reliable processing) Select Next. Step 5: Create connections On the Connections tab, create the Azure Service Bus connection: o Select Connect next to the Service Bus connection. o Provide your Service Bus connection string or select the managed identity authentication option. For managed identity (recommended), make sure your logic app’s managed identity has the Azure Service Bus Data Receiver role on the Service Bus namespace. 2. Select Next. Step 6: Configure parameters On the Parameters tab, provide values for the workflow parameters: Parameter Description Example value Azure Service Bus queue name The queue to monitor for incoming messages invoice-queue Maximum batch size Number of messages per batch (1-100) 10 Flat file schema name XSD schema name in Artifacts/Schemas Invoice.xsd Default CorrelationId Fallback value for messages without CorrelationId NO_CORRELATION_ID Target workflow URL HTTP endpoint of the downstream workflow https://your-logicapp.azurewebsites.net/... Target workflow timeout HTTP call timeout in seconds 60 Enable sequential processing Maintain message order true 2. Select Next. Step 7: Review and create On the Review + create tab, review all the provided information. Select Create. When the deployment completes, select Go to my workflow. Step 8: Upload the flat file schema Navigate to your logic app resource in the Azure portal. On the sidebar menu, under Artifacts, select Schemas. Select + Add and upload your Invoice.xsd. Confirm the schema appears in the list. Notice that: for this scenario we are using the Invoice.xsd schema, you can/must use the schema your scenario needs. Step 9: Verify and test On the workflow sidebar, select Designer to review the created workflow. Verify all actions are configured correctly. Send test messages to your Service Bus queue with different CorrelationId values. Monitor the Run history to verify successful execution and aggregation. For more information on creating workflows from templates, see Create workflows from prebuilt templates in Azure Logic Apps. Conclusion The Aggregator pattern is a cornerstone of enterprise integration, and migrating it from BizTalk Server to Azure Logic Apps Standard doesn’t mean starting from scratch. By using this template, you can: Reuse your existing XSD flat file schemas directly from your BizTalk projects Replace BizTalk Correlation Sets with CorrelationId-based message grouping via Azure Service Bus Deploy in minutes from the Azure portal templates gallery Scale elastically with Azure App Service Plan Monitor with Azure-native tools instead of the BizTalk Admin Console The template is open source and available at: GitHub PR: Azure/LogicAppsTemplates#108 Template name in Azure portal: “Aggregate messages from Azure Service Bus by CorrelationId” Source code: GitHub repository Whether you’re migrating from BizTalk Server or building a new integration solution from scratch, this template gives you a solid, production-ready starting point. I encourage you to try it, customize it for your scenarios, and contribute back to the community. Resources BizTalk Server Aggregator SDK sample Create workflows from prebuilt templates in Azure Logic Apps Create and publish workflow templates for Azure Logic Apps Flat file encoding and decoding in Logic Apps Azure Service Bus connector overview BizTalk to Azure migration guide BizTalk Migration Starter toolThe Hidden Architecture of Nano Architectures
Why does the same prompt, on the same checkpoint, with temperature set to zero, sometimes produce a different answer only when the system is under real load? If you have ever watched token three flip and then watched the whole completion diverge, you already know this is not a product bug. It is a systems fact. Here is the thing. In production, you did not deploy a model. You deployed a runtime that selects an execution plan under constraints. The weights are inside that plan. The behavior is the plan. I’m Hazem Ali — Microsoft AI MVP, Distinguished AI and ML Engineer and Architect, and Founder and CEO of Skytells. I’ve built and led engineering work that turns deep learning research into production systems that survive real-world constraints. I speak at major conferences and technical communities, and I regularly deliver deep technical sessions on enterprise AI and agent architectures. If there’s one thing you’ll notice about me, it’s that I’m drawn to the deepest layers of engineering, the parts most teams only discover when systems are under real pressure. My specialization spans the full AI stack, from deep learning and system design to enterprise architecture and security. A rule I repeat in every serious review is simple. If you cannot explain the runtime, you do not understand the model you deployed. — Hazem Ali This is the next layer after my earlier deep dive on memory, KV cache, paging, and trust boundaries in The Hidden Memory Architecture of LLMs I also break down the memory-and-paging failure modes in When Your LLM Trips the MMU This one goes lower, into the execution that decides which math actually runs. When I Had to Prove It Live I still remember the first time I had to make this concrete in front of a room full of engineers. It was during a technical session I gave, and the question came up in the exact form you’ve probably heard before: Why does the same prompt on the same checkpoint, with temperature set to zero, sometimes produce a different answer only under real load? So I answered it the only way that holds up in a serious engineering room. I didn’t frame it as randomness. I framed it as execution. Not because it sounds cleaner, but because it is the only framing that survives scrutiny: under load, the system is not evaluating the same computation. In production, you don’t deploy weights in isolation. You deploy a runtime that selects an execution plan under constraints. Under load, the constraints change at token cadence: microbatch membership shifts, shapes shift, workspace feasibility tightens, and kernels or algorithms that were legal in the calm regime can become infeasible in the pressured regime. The runtime stays correct by contract, but it executes a different plan. And once the executed plan changes, reduction staging can change. When reduction staging changes, rounding happens at different points. That can move last bits. In decoding, last bits can become different tokens when early logit margins are thin. After the first token flips, divergence is expected because the context is different. That’s what I mean throughout this article when I say: The weights are inside the plan, but the behavior is the plan. What is Happening in Runtime Let’s start with the part most teams skip: the runtime pipeline from admission to a token. A production LLM server is not a function call. It is a control plane. And under real load, it behaves like one. It is not asking “what does the model say.” It is asking “what can I execute right now without breaking my guarantees.” Right now matters. Not in theory, in milliseconds. Because every decode step is a new scheduling event. The system does not commit to a single plan for the entire completion. It keeps re-evaluating feasibility as state shifts. What can I execute at this moment, with the VRAM I still have, on the hardware state I am currently in, while staying inside isolation boundaries and latency targets. That question is not answered once per request. It is answered repeatedly, at token cadence. The queue changes. The batch changes. Memory headroom changes. Cache residency changes. Workspace availability changes. The set of legal kernel and algorithm choices changes with them. And that is the point most people miss. The runtime is not just running your weights. It is continuously selecting an execution plan under constraint. The weights are inside that plan, but behavior lives in the selection. That selection is layered. Admission shapes the effective request. Scheduling forms the batch for this step. Kernel and algorithm choice binds the math that will actually run. Memory residency and allocation decide what is feasible. Isolation rules decide what sharing is allowed. Each layer contributes to the final plan, and the plan is what you are deploying. Admission and shaping Before your prompt ever reaches the model, it gets shaped. Truncation, policy injection, tool schema expansion, routing metadata, tenant tags, prefix reuse decisions, and safety transformations. If you do not know what I mean by effective request, I mean the exact token sequence that the model saw after shaping. That is the only input that matters for reproducibility. Batching and step level scheduling Modern servers do not just batch requests. They batch token steps. In a continuous batching system, token step timing feeds back into batching decisions. A slightly slower step changes who joins the next step. Who joins the next step changes shapes. Shapes change kernels. Kernels change numeric pathways. This is not an opinion. It is why vLLM exists. The PagedAttention paper describes serving as a batching problem where KV cache grows dynamically, wastes memory through fragmentation, and limits batch size. It introduces block level KV management and builds vLLM on top of it as an LLM serving system. Kernel plan selection and library behavior Once shapes are known, the runtime selects kernel variants and library algorithms that are feasible for those shapes and the workspace currently available. This is the part people underestimate. The same operator can have multiple valid implementations. The chosen implementation can change when workspace is tight, when shapes change, or when the engine wants to trade latency for throughput. Memory allocation and residency KV cache, activations, temporary buffers, workspace, graph memory, and communication buffers compete for VRAM. Under pressure, allocation patterns change. Fragmentation changes. Residency changes. Cache locality changes. All of that changes the system timeline and the feasible plan space. If you want a one line summary that is accurate in 2026 production inference, it is this. Inference is a scheduling problem plus a memory residency problem, and the model is inside that. The Scope First, Let me put it very clear. I am not claiming every deployment is nondeterministic. I am not claiming every kernel variant flips tokens. I am not claiming seeds are useless. I am making a narrower claim, the kind you can defend in an incident review without hand waving. Floating point math is not associative. Order matters. When you parallelize, you change the order of operations, and it is therefore valid for parallel results to differ from a sequential evaluation. NVIDIA states this directly in the CUDA C Best Practices Guide. CUDA also makes a foundational guarantee to the hardware and scheduler, not to your intuition. Thread blocks must be able to execute independently, in any order, in parallel or in series. That freedom is part of the programming model, not an edge case (ref). Now connect those two facts. If accumulation order changes, the last bits can change even when every operation is correct, because floating point addition is not associative. NVIDIA explicitly calls this out as well. Then layer in what serving stacks actually do. Production systems intentionally reshape execution through continuous batching and KV memory management. vLLM is a published example of this co design, where serving throughput is achieved by dynamic batching and memory-aware KV handling. Finally, bridge the nano to the semantic. When early logit margins are small, tiny numeric deltas can reorder the top candidates, and a single token flip is enough to diverge the entire completion. Here is the part that should feel a little scary, because it changes what you think you are operating. Under real load, the system is not just slower. It can enter a different execution regime. Batch composition shifts, shapes shift, workspace and residency shift, and the runtime is forced into a different set of legal kernel and algorithm choices. Nothing “breaks.” No bug is required. The system is still correct by contract. But your output is now a property of the regime you are in, not the demo you validated. That means you can pass every determinism test at idle and still ship a system that drifts only when it matters, at p95 and p99, when queues are long and memory headroom is tight. The first time you notice is often a user screenshot, an audit question, or an incident report where two replicas disagree on the same request because the runtime state was not the same. The equation principals should use in incident reviews Most teams ship with the demo mental model. y = f(x, θ) One prompt in, one checkpoint, one output. If the output changes, someone concludes the weights changed, or “AI is random.” That is not how production inference behaves, because production inference is not just a function. It is execution under constraint. Production behavior is closer to this. y = Decode( Exec(θ, x; s) ) θ is still the same weights. But the thing you actually shipped is Exec, and Exec is chosen. It is chosen per step, under the current state of the system. The behavior you observe is the behavior of the executed plan, not the abstract weights. X is not the prompt. X is the effective request. X is the exact token sequence the model saw after shaping. Truncation, policy injection, tool schema expansion, routing metadata, prefix reuse, safety transforms. All of that can change what the model actually receives. If you cannot reconstruct x, you are not replaying the request. You are replaying an approximation. Here is the minimum you should log for x, even if you cannot store raw text: # minimal "x" record: enough to reproduce or prove you cannot trace_x = { "req_id": req_id, "raw_prompt_sha256": sha256(raw_prompt), "effective_text_sha256": sha256(effective_text), "effective_tokens": len(effective_tokens), "truncated": truncated, "trunc_reason": trunc_reason, # e.g., "latency_guard", "context_cap" "decode_cfg_applied": decode_cfg, # temperature/top_p/max_tokens, etc. "shaping_events": events, # ["policy_inject:v3", "tool_schema:v2", ...] } S is not a vibe. S is the execution state that decides the math. S is what principals should demand in a postmortem, because this is what turns “it drifted” into “this plan executed under this regime.” At minimum, s includes: per-step batch composition and shape class queue delays and scheduling outcomes VRAM headroom and workspace availability cache pressure signals precision path and engine fallbacks distributed timeline signals (TP/PP latency, collective stalls) isolation posture (what batching is allowed) Why this matters: in continuous batching, time becomes part of semantics. A few milliseconds of delay changes who gets co-scheduled at the next token step. That changes shapes. Shapes change kernel/algorithm feasibility. Feasibility changes the numeric pathway. When early logit margins are thin, a tiny pathway delta is enough to flip the argmax. Here is a short, practical “s” record you can emit per decode step: # per-step "s" record: what plan ran, under what pressure step_s = { "req_id": req_id, "step": t, "batch_fp": sha256(",".join(sorted(batch_req_ids)))[:12], "shape": f"q=1,k={klen},h={heads},d={hidden},tp={tp}", "queue_ms": queue_ms, "gpu_ms": gpu_ms, "vram_free_mb": vram_free_mb, "workspace_free_mb": workspace_free_mb, "kv_regime": kv_regime, # "normal" | "pressured" | "paged" "precision_path": precision_path, # "bf16" | "fp16" | "tf32" | "fp32" "algo_id": algo_id, # backend/engine specific "kernel_variant": kernel_variant, # if available "isolation_mode": isolation_mode, # "shared" | "strict" } The incident-review translation If you only ask “what prompt did the user send” and “what weights did we run,” you are using the demo equation. You will argue about seeds, debate “randomness,” and never converge. The production equation forces the real question. Which plan executed, under which constraints, and what state pushed us into that plan. The line principals should repeat until teams internalize it is simple. Weights are static. Behavior is a property of the executed plan. And the executed plan depends on state. If you want one more operational layer that makes this feel real, add a regime marker. Regime changes are where “stability” collapses without any bug: def regime(vram_free_mb, paging_on, isolation_strict, queue_p95_ms): if isolation_strict: return "isolation_strict" if paging_on: return "paging" if vram_free_mb < 1024: return "memory_pressured" if queue_p95_ms > 50: return "queue_degraded" return "normal" When the regime changes, the feasible plan space changes. When the plan space changes, the executed math can change. That is the production reality your incident review must be able to explain. Floating point order is where small deltas are born Let’s break it down without hand waving. Finite precision makes rounding part of the computation Floating point math is not real-number math. Every add and multiply is followed by rounding to the representable format you are using. That rounding is not “noise.” It is part of the computation. Once you accept that, one consequence becomes unavoidable. Order matters. NVIDIA states the rule clearly: floating point involves rounding, and when you parallelize you can change operation order, so parallel results may not match sequential results. Why LLM inference is a perfect storm: reductions everywhere Now connect that to what an LLM does at inference time. LLM inference is reduction-heavy by design. Dot products in GEMMs, attention score accumulation, softmax normalization, layer norm statistics, even top-k selection pathways. These are not single operations. They are many partial operations combined into a final scalar or vector. In floating point, the way you combine partials is the outcome. GPU reductions are staged: partial sums, then merges A reduction on GPU is not “a sum.” It is a staged reduction of partials. On a CPU, you can imagine a left-to-right accumulation: ((((a1 + a2) + a3) + a4) + ...) On a GPU, that mental model is wrong. The GPU is built to run thousands of threads. So it computes partial sums in parallel and then merges them in stages. The staging pattern is determined by kernel design and how the backend maps the problem to hardware. Put the figure here, right after the staging idea lands. The staging depends on decisions you do not control at the prompt layer: how data is tiled into blocks how each block maps to warps how many partials each warp reduces whether it uses warp-level primitives, shared memory, or tensor core fragments how the final merge is staged across blocks Change the tile size, or the block shape, or the occupancy, and you often change the staging order. Change the staging order, and you change when rounding happens. You can get two results that are both correct under IEEE floating point rules, and they differ in the last bits. This is not a bug. It is the contract of finite-precision parallel math, applied at scale. Why the last bits move at the core level Floating point addition is not associative under rounding because rounding happens after each operation. The error introduced at each step depends on the magnitude and sign of what you are adding at that step. When you change the staging order, you change: which numbers get added together early which partial sums get rounded early how cancellation behaves when positive and negative terms interact when large and small magnitudes meet, where small values can lose representable impact That is the core mechanism behind “small deltas.” It is not mystical. It is mechanical. Why this shows up in production serving, not in your demo LLM inference is dominated by massive matrix operations and attention. Under the hood, those paths accumulate across large dimensions. An accumulation is exactly where rounding order matters most. And the server does not always run the same kernel variant for those ops. Under load, shape shifts and workspace pressure can push the backend into different implementations. Different implementations often imply different tiling. Different tiling implies different staging. Different staging implies different rounding. Different rounding implies different last bits. So even with an identical prompt, identical checkpoint, and temperature set to zero, you can still see tiny numeric differences when: batch composition changes and produces different effective shapes the engine picks a different algorithm because workspace is tighter the kernel selects a different tile path due to shape class and occupancy the GPU is in a different pressure regime, changing feasibility and scheduling behavior Those deltas are small, but they are real. And in decoding, small can be enough. The bridge from ulps to language: logits, argmax, divergence A tiny last-bit difference is often irrelevant, Until it hits a decision boundary. At decode step t, greedy decoding chooses an argmax. If the top logits are close, a small delta can swap the ordering. Once token t changes, the context changes, and the completion diverges. That is not randomness. That is deterministic branching from a slightly different numerical pathway. So the actionable takeaway is not “GPUs are nondeterministic.” It is this. Parallel math is allowed to produce multiple correct last-bit outcomes, and LLM decoding can amplify those outcomes into different text when margins are thin. CUDA scheduling makes ordering a form of runtime state CUDA makes a stronger statement than most people realize. Thread blocks must be able to run independently. It must be possible to execute blocks in any order, in parallel or in series. That is why the same kernel can execute with different inter block ordering depending on occupancy, contention, and scheduling. Now bring atomics into the picture. Atomics guarantee correctness of each update. They do not guarantee the arrival order of updates across threads and blocks. When floating point updates arrive in different legal orders, the final sum can differ in the last bits, because floating point addition is not associative. If you do not know what atomic add means, here is the useful definition. Atomic add ensures updates do not overwrite each other. It does not ensure which thread gets there first. This is the nano architecture layer that explains a lot of weirdness. Many engineers assume determinism is a property of weights. In practice, determinism is constrained by the legal reorderings of parallel execution. Logit margin is the bridge from ulps to language Now we connect the last bits to a changed sentence. At decode step t, greedy decoding picks the argmax over logits. Let the top two logits be ℓₐ and ℓ_b. Define the margin: mₜ = ℓₐ − ℓ_b A token flip happens when a small perturbation changes the ordering of these top two. If you want an operational translation, it is this. If the model barely prefers token A over token B, a tiny numeric delta can make it prefer B. Once token t changes, the rest of the completion evolves under a different context. Divergence is expected. This is why I keep pushing one instrumentation idea that sounds boring until you need it. Measure early step margins. You cannot manage stability if you never measure how close the decision boundary is. The effective request problem, the quiet killer of reproducibility Here is the pattern I see in almost every serious production investigation. The team replays the user prompt, cannot reproduce the output, and concludes the model is nondeterministic. Then the incident dies in ambiguity. And then, usually too late, someone asks the only question that matters. What did the model actually see. “In every postmortem, I ask one question before I look at weights, kernels, or seeds: what did the model actually see. If we cannot answer that, nothing else is evidence.” - Hazem Ali In production, the user prompt is not the input. It is an ingredient. By the time a request reaches the model, it has passed through a shaping pipeline that exists to keep the system safe, fast, and multi-tenant. That pipeline is not cosmetic. It can change semantics, length, and even decode behavior. The result is the only input that matters for reproducibility. The effective request. This is the same thesis you have already accepted earlier in the article. y = Decode( Exec(θ, x; s) ) If you do not know x, your replay is not valid. If you do not know s, your replay is not comparable. And if you only log the raw prompt, you are logging neither. Shaping changes semantics, not just length Truncation is the obvious one. Under load, systems often cap context length to protect latency and GPU memory. Same prompt, different truncation boundary, different effective context, different output. Nothing “random” happened. You executed a different input. But truncation is only the beginning. Policy injection can prepend or append system text that changes intent. Tool schema expansion can add hundreds or thousands of tokens and push the request over a context boundary. Routing metadata can select a different template. Prefix caching can reconstruct parts of context from cached state rather than raw text. Safety transformations can rewrite or neutralize content. Even small differences here can shift early logits when margins are thin, and this article already showed how small deltas become different tokens. The worst part is that this is silent by default. The user sees their prompt. Engineers see the prompt in logs. The model sees a different token sequence. Then everyone argues about reproducibility using the wrong input. Why this interacts with load, not just correctness Under low load, your system often has enough headroom to be generous. Longer context, fewer cutoffs, stable routing, more consistent batching, and fewer fallbacks. Under real load, shaping becomes defensive. Dynamic truncation thresholds kick in. Tool schema expansions collide with context limits. Prefix reuse behavior changes. Safety gates can become stricter. The same user text can produce a different effective request, and therefore a different output, precisely when the system is under pressure. So if you are only validating reproducibility at idle, you are validating a different system than the one you ship. What principals should require in telemetry If you want strict reproducibility, you must log the execution contract per request. Not the story. The contract. At minimum: effective token count after shaping truncation boundary and reason final merged decode config actually applied policy gates that modified prompt or decode path whether prefix cache was used, and what cache key was referenced routing template version and system message hash If you are privacy constrained, you still can log hashes and structural facts. You do not need raw prompts to diagnose effective request drift. You need verifiable fingerprints. Here is the short version in one line. If you only log the user prompt, you have not logged x. You have logged an approximation of x. And without x, you cannot claim reproducibility. You can only hope for it. Continuous batching, why time becomes part of semantics This is where principal level thinking matters. Continuous batching does not just increase throughput. It changes the execution context at each token step. Batch composition changes shapes. Shapes influence kernel selection and workspace feasibility. Those choices can change reduction structure and rounding pathways. If you want a published anchor, use vLLM. The PagedAttention paper frames high throughput serving as a need to batch many requests, but KV cache grows dynamically and wastes memory through fragmentation. It proposes PagedAttention and builds vLLM on top of it, with block level memory management and flexible sharing of KV cache to reduce memory usage. (arxiv) Here is what this really means in production. The server is selecting which requests share a step. That changes the math shapes. That changes the executed plan. That is why the same prompt behaves differently under load even at temperature zero. Algorithm selection and engine fallback The hidden variability people forget about If you have ever tried to reproduce a drift across replicas and felt like you were chasing ghosts, this is usually the layer you were missing. Libraries and engines choose, Not in a philosophical sense. In a literal, per-operator, per-shape sense. The same attention call is a fork in the road between multiple legal tactics, each with different tiling, different reduction staging, different fusion boundaries, and different temporary memory requirements. Your checkpoint is the same, your prompt is the same, your temperature is zero, and the output still moves because the executed plan moved. PyTorch says the quiet part directly. Disabling cuDNN benchmarking makes cuDNN deterministically select an algorithm, and PyTorch stresses this is different from the deterministic setting. That is the whole story in one sentence: one switch affects how the backend selects an algorithm, another affects whether the selected algorithms are deterministic. Those are separate layers, and under load they can diverge. Now go down to the core of the core. A tactic is not fast or slow. In production serving, a tactic is legal or illegal under the constraints of this token step. The constraint that forces most plan switches is not compute. It is workspace feasibility. Many high-performance kernels need scratch buffers. Some need enough contiguous space to stage tiles, reorder operands, hold partials, or run fused epilogues. When VRAM is fragmented or headroom drops, a tactic becomes impossible even if it is the tactic you validated at idle. The engine does not throw a warning. It simply selects another legal tactic. That is the first uncomfortable point. The second uncomfortable point is what makes this align perfectly with the next section. The constraint is not only “how many MB are free.” The constraint is the memory hierarchy state of the chip. Under load, two replicas can have the same free VRAM and still be in a different regime because the chip is not one pool of memory. It is HBM plus an on-die L2, plus TLBs, plus page tables, plus a fabric that is arbitrating traffic between SMs, L2 slices, and HBM controllers. When that hierarchy shifts, latency per token step shifts. And in continuous batching, a few milliseconds is not a timing detail, it is a scheduling input. This is how a performance event becomes a behavior event without any bug. The engine’s planner sees a world where a tactic that was “best” at idle is no longer best, or no longer feasible, because the chip is in a different pressure state. Your runtime is still correct. It is just operating a different plan in a different regime. One op, multiple legal kernels. The chosen tactic depends on shape class and feasibility. Now bring TensorRT into the picture, because it makes the precision dimension explicit. TensorRT states TF32 Tensor Core usage is not guaranteed and it can fall back to FP32, and it documents configuration controls around TF32. That statement is not about “precision preference.” It is about the reality that precision is part of tactic selection. Precision changes which instructions execute and how accumulation is staged. When your early logit margins are thin, a small pathway delta can swap the argmax at one step. One token flips, and the rest of the completion deterministically diverges. So “temperature zero” is not a determinism guarantee. Temperature governs sampling. It does not pin the execution pathway. If you want a more mechanical anchor, treat matmul the way NVIDIA exposes it: cuBLASLt has a preference descriptor for applying algorithm search preferences and fine-tuning the heuristic function. That is not marketing. That is the API admitting that algorithm selection is a constrained search problem. Now the part that gets rare, and the part most teams never write down. CUDA’s programming model requires that thread blocks be able to execute independently and may execute in any order, in parallel or in series. This matters here because tactic switches often change block geometry and tiling. Different block geometry changes reduction staging. Reduction staging changes where rounding happens. Even if every operation is correct, last bits can move because you legally changed the staging of partial sums. You do not need randomness. You need a different legal staging tree. Now pull security into the same frame, because it is not a separate layer in production. Security posture changes what the scheduler is allowed to do. Isolation constraints reduce batching freedom. Reduced batching freedom increases tail latency. Tail latency pushes you toward tighter admission controls and more aggressive memory behavior. That shrinks the feasible tactic set sooner. In other words, security decisions can move you across regime boundaries faster, which increases plan switching frequency. Stability becomes an SLO dimension of your security posture, not a property of your weights. This is the business consequence that shows up in the worst possible way. So here is the operational rule I use in reviews. If you cannot prove which plan ran, you cannot claim reproducibility. And that leads to the only practical addition that belongs in this section before we move into VRAM bandwidth and cache residency. VRAM bandwidth, cache residency, and why memory hierarchy becomes control plane input Let’s talk about the performance facts that quietly become behavior facts. And yes, I know how complex this gets. I have watched strong staff and principal engineers get lost here, not because they are weak, but because the system crosses too many layers at once: GPU microarchitecture, allocator behavior, kernel tactics, batching policy, and SLO-driven control loops. No single dashboard shows you the full causal chain. That is exactly why I frame it this way. It is not “performance tuning.” It is a coupled control system. So let me break it down cleanly, from the chip outward, until the behavior change becomes inevitable. NVIDIA describes H100 SXM5 as having HBM3 bandwidth around 3 TB/s and an L2 cache of 50 MB designed to reduce trips to HBM by caching repeated accesses. Most teams read that as “the GPU is fast.” In serving, it is more precise to say: the GPU gives you a memory hierarchy with regimes, and your runtime is forced to adapt to whichever regime you are currently in. The chip-level model you should carry in your head Decode is not one big matmul. It is a loop that repeatedly touches a shifting working set: KV blocks for the active sequences attention metadata (block tables, indirection, masks) sampling buffers (logits, top-k/top-p structures) runtime bookkeeping for continuous batching Those accesses are not purely streaming. They are pointer-heavy, and their locality depends on how your KV is laid out, which requests are co-scheduled, and how fragmented your memory becomes under churn. Here is the simplest mental model that is still honest: B_HBM is the number of bytes actually read from HBM during this step. B_L2miss is the number of bytes that missed L2 and therefore had to be fetched from HBM. t_translate is the address-translation tax: extra time from TLB misses and page-table walks. That last term is the one that surprises people. It’s “invisible” until it dominates. Why L2 residency becomes a control-plane input Now connect that to decode, Decode repeatedly reads KV state. If L2 hit rate drops, HBM traffic rises. When HBM traffic rises, stalls rise. When stalls rise, token-step latency shifts. When token-step latency shifts, the server changes batching decisions. This is the control loop you should keep in your head: L2 hit rate ↓ → t_step ↑ → Δt ↑ → batch composition changes → shape class changes → tactic set changes That is the bridge from “cache miss” to “different plan executed.” In continuous batching, time is not just an output metric. Time is an input into the next scheduling decision. A few milliseconds can change who gets co-scheduled at the next token step. That changes shapes. Shapes change feasible kernels and algorithms. That changes the executed math. And if early logit margins are thin, a small pathway delta can flip a token and send the rest of the completion down a different branch. Rare but matters: the translation tax that breaks the “free VRAM” illusion Two replicas can report similar free VRAM and still be in different regimes. Why? Because the chip is not “a pool of memory.” It is an on-die cache, translation structures, page tables, and a fabric that is arbitrating traffic under pressure. When KV is stored in blocks (or pages) and those blocks are scattered due to churn, you often get: worse spatial locality more distinct memory regions per step more TLB pressure more page walks Page walks are not abstract. They are memory reads. They compete with your payload reads. Under real load, this turns into self-inflicted HBM traffic. So you can be “bandwidth rich” on paper and still be “latency poor” in practice because the working set became translation-hostile. This is how a performance event becomes a behavior event without any bug. A concrete KV bandwidth sanity check If you want a back-of-the-envelope check for why decode becomes memory-shaped, use a conservative estimate. Per token step, you often need to read a large portion of KV for the active context. A rough model is: KV bytes per step ≈ 2 × B × L × H × D × s Where: B is batch size (number of sequences co-scheduled in the step) L is current context length (tokens already in KV) H is the number of attention heads (or KV heads, depending on the model) D is head dimension s is bytes per element (2 for fp16/bf16, 1 for int8, etc.) The factor 2 accounts for K and V. Even if your kernel is compute-efficient, you are still moving a lot of bytes. If locality collapses and L2 misses rise, you shift into an HBM-limited regime fast. That is the mechanical reason your p95/p99 step time moves under load, even with the same checkpoint and temperature. Business impact, stated plainly This is why drift shows up where it hurts: p95 and p99. At idle, L2 residency is generous, fragmentation is lower, translation pressure is calmer, and step time is stable. Under load, residency collapses, translation tax rises, allocator feasibility tightens, step time stretches, and your control plane adapts by changing batching and shapes. That can move you into different execution plans without any model change. An enterprise buyer does not care whether you call it “L2 miss driven plan churn.” They care that two identical requests disagree and you cannot explain it. So the takeaway I want principals to internalize is simple: In continuous batching, memory hierarchy state is control-plane state. It shapes latency. Latency shapes batching. Batching shapes shapes. Shapes shape feasibility. Feasibility shapes the executed plan. That is how “performance” becomes “behavior.” Multi node tensor parallel, the execution plan extends across the fabric Once you go multi-node tensor parallel, you add a second execution plane that most teams underestimate. You are no longer operating only a GPU runtime. You are operating a distributed timeline. And the timeline is not a background detail. In continuous batching, the timeline becomes a control input that reshapes batching, shapes, and eventually the executed plan. Let me be precise about what I am claiming, and what I am not. I am not going to claim collectives reorder arithmetic inside a kernel. That would be sloppy. The correct claim is this: Distributed synchronization changes the timeline. The timeline changes admission and batching. Batching changes shapes. Shapes change which plans are legal. That’s enough to explain why the “same prompt, same checkpoint, temp=0” can drift only under real load. The minimal equation you should carry At each decode step, your latency is no longer “GPU time.” It’s GPU time plus fabric time: t_step ≈ t_compute + t_comm + t_sync And the part that hurts is that t_comm and t_sync are not stable. They are affected by contention, queueing, stragglers, and topology. A useful mental model for the communication piece is the classic latency–bandwidth form: t_comm(message) ≈ α + (n / β_eff) α is the per-collective startup and synchronization overhead n is bytes moved β_eff is the effective bandwidth you actually get under contention In isolation, this looks like performance math. In a continuous batching server, this becomes behavior math, because t_step feeds back into the next scheduling decision. What actually happens in multi-node TP at token cadence Tensor parallelism shards the model across devices. Every token step requires cross-device coordination for some portion of the layer execution. In practice, this means collectives become part of the critical path. NCCL’s collective ops are explicit about the semantics: for example, AllReduce reduces values across ranks and returns identical results to all ranks. That tells you what the runtime must do: it must wait for coordination across ranks before progressing. So the decode loop becomes: execute local compute for this step hit a collective boundary wait for the slowest rank to finish and for the fabric to deliver proceed That “slowest rank” detail is the piece people feel but rarely name. In distributed inference, p99 is often a straggler story. A single congested link, a slightly delayed rank, or a transient fabric stall turns into a global stall because collectives synchronize progress. In other words, a multi-node TP system behaves like a coupled oscillator: the fastest GPU is still gated by the slowest collective. Why this changes the executed plan, not just the latency Here’s the bridge to the thesis of the whole article. In a continuous batching server, you do not just execute requests. You continuously reform microbatches at token cadence. That means step time affects who joins the next step. And in multi-node TP, fabric jitter is one of the biggest sources of step-time variability. So when comm jitter shifts t_step, it shifts the schedule: queue delay changes microbatch membership changes effective shape class changes workspace feasibility changes tactic choice changes You already established earlier that a changed shape class can force a different tactic set. Multi-node TP adds a new reason shape churn happens: not only GPU pressure, but fabric timing pressure. So the claim stays clean and defensible: Distributed synchronization doesn’t need to change arithmetic to change behavior. It only needs to change the timeline that drives batching. Chip-to-fabric reality: why infrastructure details belong in the reproducibility record At this scale, the infrastructure is part of the runtime. According to Azure Docs, Azure’s ND H100 v5 series is explicitly positioned for tightly coupled scale-up and scale-out Generative AI and HPC workloads, and it’s built around the idea that the fabric matters, not just the GPUs: If you are running multi-node TP in production, treat fabric telemetry as part of your reproducibility record. Not because it is fun. Because it changes the system timeline that drives batching. A practical minimum is to track per-step: collective type on the critical path (e.g., all-reduce / all-gather) comm time and jitter (p50/p95/p99 per step window) rank skew (max(rank_time) − min(rank_time)) effective bandwidth estimate (n / t_comm) retransmit / congestion signals if your stack exposes them a “fabric regime” marker: normal vs congested vs degraded When drift becomes expensive This is one of the reasons enterprise teams report the most confusing failures only at load. At idle, your timeline is stable, your microbatches are stable, your shapes are stable, and your plan selection is stable. Under real load, the fabric introduces jitter, jitter reshapes batching, batching reshapes shapes, and shapes reshape the executed plan. Now two replicas can disagree, not because the model changed, but because the timeline differed. That shows up as: inconsistent answers across replicas in high-stakes workflows reproducibility failures during audits and incident reviews “regressions” after scaling out, even with the same checkpoint and code support costs and credibility loss because you cannot explain why behavior changed only at p95/p99 So the operational sentence I want you to carry into your postmortems is: In multi-node tensor parallel inference, the execution plan extends across the fabric. If you do not log the fabric timeline, you are missing part of the runtime state that decides which plan was feasible. Where Infrastructure Stops Being “Just Infrastructure” Once you accept the thesis of this article, one conclusion becomes unavoidable: cloud choices are not just cost and convenience decisions. They shape which execution regimes your runtime will enter under pressure. At scale, you are no longer buying “GPUs.” You are buying: A fabric and topology that holds up under synchronized token-step collectives A VM family with predictable characteristics for tightly coupled scale-out workloads (the kind multi-node inference actually is) An isolation posture that can be enforced in hardware when your threat model requires it, without hand-waving away the runtime implications First-class observability for GPU behavior, not just CPU and request traces, so you can correlate drift with the state variables that caused it (for example, exporting NVIDIA DCGM metrics into managed Prometheus and Azure Managed Grafana on AKS). This is the quiet reason certain platforms feel “more stable” in production. Not because the model is different, but because the runtime state is easier to constrain, measure, and explain when the underlying infrastructure is designed for the exact regime you’re operating in. Quantization effects on execution paths and causal stragglers in multi-node TP Let me be direct about what most articles miss when they discuss distributed inference at scale. The conversation typically stops at "how many GPUs" and "what's the bandwidth." That's not wrong. It's just incomplete. What's missing is the interaction between quantization-induced plan churn and straggler amplification in the collective path, two forces that quietly reshape your execution regime under VRAM pressure and fabric contention. These are not theoretical curiosities. They are production realities at 100+ GPU scale, the kind of scale where you can no longer afford to treat quantization as a "precision choice" or stragglers as a "latency outlier." At that scale, they become causal inputs to your runtime's decision surface. Quantization variability: not just precision, but plan selection When teams talk about INT8 or FP8 quantization, the conversation usually centers on memory savings and throughput gains. That's the marketing layer. The execution layer is more nuanced: quantization changes which kernels are legal, where fusion boundaries land, and how reduction trees are staged. Here's what I mean in concrete terms. Under VRAM pressure, your serving stack may need to requantize activations mid-forward-pass to stay within memory bounds. That requant step is not "free" in the plan sense. It introduces: dequant/requant cycles that break fusion opportunities you had in the FP16 path new non-associative operations in the reduction tree, where rounding happens at different stages fallback paths when the quantized kernel variant lacks workspace or doesn't support the current shape class Let me state this in the language of the article's thesis: quantization is not a data type. It is a tactic constraint that reshapes the feasible plan space. Memory pressure can force dequant/requant cycles, change fusion boundaries, and trigger fallback kernels with different reduction staging, producing last-bit differences that can flip tokens during decoding. The practical consequence? Two replicas running "the same quantized model" can execute different kernel variants when one is memory-pressured and the other is not. The memory-pressured replica may be forced into a fallback path with different reduction staging. Different staging means different rounding order. Different rounding order means different last bits. And in decoding, last bits can become different tokens. I've watched incident reviews where teams assumed INT8 was "deterministic" because they set the quantization scheme once at export time. What they missed is that the runtime's quantization pathway depends on the state of VRAM fragmentation, workspace availability, and kernel preference histograms, exactly the regime-dependent variables we've been building toward throughout this article. If you're operating at scale, instrument this. Track: per-step kernel selection via cuBLASLt preference descriptors dequant/requant cycle counts when memory pressure rises fallback events when preferred quantized tactics become infeasible whether the executed plan matched the "expected" quantization pathway This is rare telemetry. Most teams never see it because they're not running large enough clusters under sustained pressure. But once you cross into 100+ GPU inference workloads, quantization-induced plan churn becomes visible in your p99 drift signatures. Causal stragglers: when one rank's fallback stalls the collective Now let's talk about the fabric-scale pathology that couples with everything we just discussed: head-of-line blocking in distributed tensor parallelism. You already know from the multi-node TP section that collectives synchronize progress. The fastest rank waits for the slowest. That's the contract. What's less documented—and what I've only seen formalized in internal NVIDIA serving postmortem templates—is how a single rank's kernel fallback can become a collective-wide straggler, and how that straggler amplifies through the batching feedback loop. Here's the causal chain: One rank enters memory pressure. Maybe fragmentation is worse on that device, maybe it's handling a slightly different KV layout due to request assignment. That rank falls back to a slower tactic. The preferred kernel requires workspace. Workspace isn't available. The engine selects a legal fallback. The fallback kernel takes longer. Not by seconds—by milliseconds. But in a collective, milliseconds matter. The collective waits. AllReduce can't proceed until all ranks contribute. The straggler becomes the bottleneck. Step time stretches. The stretched step reshapes the next batch in continuous batching. Different batch, different shapes, different feasibility. The cycle repeats. Now multiple ranks may be in fallback paths. The p99 drift you're seeing isn't random—it's a feedback loop. This is what I call a causal straggler: not just a slow rank, but a rank whose performance degradation causally reshapes the execution regime of the entire TP group. And here's where quantization and stragglers intersect. If one rank is under more VRAM pressure and is forced into more frequent dequant/requant cycles, it becomes the straggler. Its quantization pathway differs from the other ranks—not because the model changed, but because the memory regime changed. That difference in pathway becomes a difference in step time. That difference in step time becomes a collective stall. That stall becomes a batching change. That batching change becomes a new plan. The output drifts, and you're left wondering why "the same checkpoint at temperature zero" produced different text only under load. The answer is: you weren't in the same execution regime. You were in a regime where one rank's memory pressure caused a straggler, the straggler caused a timeline shift, and the timeline shift caused a plan change. Rarity value: why this knowledge is elite production battle scars Let me be honest about why these gaps are rare. Most teams never operate at the scale where these effects dominate. If you're running inference on 8 GPUs, you might see hints of this. At 100+ GPUs with multi-node TP and continuous batching under sustained load, it's no longer a hint—it's the signature. The teams that do operate at this scale track: cuBLASLt preference histograms to detect when algorithm selection is churning across steps NCCL timeline traces to identify straggler signatures and correlate them with per-rank memory state per-rank kernel fallback events to see when one device is operating a different plan than its peers quantization pathway divergence across ranks under pressure This is the telemetry that doesn't show up in tutorials. It shows up in postmortems at hyperscaler SLO thresholds, where p99 latency violations trigger incident reviews and someone finally asks: "Why did replica 3 disagree with replica 1 only during the peak load window?" The article you're reading now covers single-node memory regimes beautifully. What bridges to 10/10 elite production knowledge is this: fabric-scale causality. The understanding that in multi-node TP, your execution regime is not just shaped by your GPU's memory state—it's shaped by the worst GPU's memory state, because collectives couple everyone's timeline. That's the gap. That's the rarity value. And if you're building or operating inference at 100+ GPU scale, that's the layer where your next outage is hiding. Peak depth: wavefront divergence, tensor core fragmentation, NCCL backpressure, and ISR collision Everything above operates at the principal and staff engineer level. What follows is the layer below that—the chip architect handoff, where you stop talking about "plans" in the abstract and start talking about warp stall cycles, tensor core fragment occupancy, NCCL retransmit chains, and memory evaporation under replication pressure. I'm writing this section because it's the part I never see published outside internal design reviews, and because these are the exact pathologies that turn a well-architected inference cluster into a system that disagrees with itself only during peak traffic. "Most engineers debug the layer they understand. The system breaks at the layer they don't. In production inference, that layer is almost always the one where microarchitecture meets scheduling meets the fabric." — Hazem Ali Wavefront divergence in decode attention kernels Let me take you inside the warp. In SIMT execution, a warp is 32 threads executing in lockstep. When all threads follow the same control path, you get full utilization. When they diverge—different threads take different branches—the warp must serialize both paths. That's textbook GPU architecture. What's not textbook is how this interacts with paged KV attention in production decode loops. In a paged KV system (the exact kind vLLM introduced), KV blocks are scattered across VRAM. Different sequences in the same microbatch may have their KV blocks in different residency states: some hot in L2, some cold in HBM, some partially evicted under paging pressure. When the attention kernel issues loads for KV blocks, threads within the same warp can stall at different rates depending on which blocks they're accessing and where those blocks reside. This creates a subtle but measurable pathology: Lane divergence inside the attention kernel. Not control-flow divergence in the traditional sense, but memory-latency divergence: some lanes return fast (L2 hit), some stall (HBM fetch), and the warp can't retire until the slowest lane completes. Register pressure amplification. When warps stall, the SM must keep their register state live. Under heavy stalling, register pressure rises, which can force the compiler to spill to local memory (which lives in L2/HBM). Spills create more memory traffic, which creates more stalls. It's a feedback loop at the microarchitectural level. Measurable p99 step variance in identical-shape batches. This is the part that confuses teams. Two consecutive decode steps with the same batch size and the same sequence lengths can have different step times, because the KV block residency pattern differed. The shape was identical. The memory topology was not. If you want to see this in practice, the tool is Nsight Systems. What you're looking for: # Nsight Systems trace analysis: partition warp stall cycles # Look for these stall reasons in the GPU metrics view: # - smsp__warps_issue_stalled_long_scoreboard → memory dependency stalls # - smsp__warps_issue_stalled_short_scoreboard → register dependency stalls # - smsp__warps_issue_stalled_no_instruction → instruction cache miss # # Correlate with: # - l1tex__t_sectors_pipe_lsu_mem_global_op_ld → global load sectors (KV fetches) # - lts__t_sectors_srcunit_tex_op_read_hit_rate → L2 hit rate during attention # # The diagnostic signal: when stall_long_scoreboard spikes correlate with # L2 hit rate drops, you're seeing KV residency divergence across warps. The stall partition tells you why the warp stalled. When you see long_scoreboard stalls dominating during attention kernels—and you see them correlating with L2 miss rate fluctuations—you're observing exactly the KV residency divergence I'm describing. The warp is waiting for scattered KV blocks, and the scatter pattern changes with every batch because paging decisions are state-dependent. This is how "identical shapes" produce different timelines. The shape is the same. The KV block map is not. And the block map is a function of runtime allocation history—the same state-dependent variable that drives everything else in this article. Tensor core fragment utilization collapse under shape churn Now let's go inside the tensor cores themselves. H100 and Blackwell tensor cores operate on matrix fragments—fixed-size tiles that map directly to the hardware's matrix multiply-accumulate units. On H100, the native fragment sizes for FP16 are typically 16×16×16 (m×n×k). When your operand dimensions align cleanly with fragment boundaries, you get full utilization. When they don't, you get fragment waste: the hardware still executes full fragments, but some of the lanes carry padding zeros. In continuous batching, shape churn is the norm. Your microbatch dimensions change at token cadence. And this is where a subtle but devastating efficiency collapse hides. Consider two microbatches that arrive one step apart: # Step t: B=16, L=2048 → GEMM shape aligns cleanly with 16×16 fragments # Fragment utilization: ~98% # cuBLASLt selects: WMMA-based kernel (tensor core native) # # Step t+1: B=17, L=2047 → GEMM shape straddles fragment boundaries # Fragment utilization: drops below 25% on trailing tiles # cuBLASLt selects: fallback to non-WMMA FP16 kernel # (or WMMA with heavy padding, depending on heuristic) The difference is one sequence in the batch and one token in context length. The performance consequence is that the runtime switches from tensor core native execution to a scalar FP16 path. That's not a minor variant. That's a fundamentally different instruction mix, a different reduction tree, and a different accumulation order. The ulp deltas that result from this switch don't stay contained in the GEMM output. They propagate forward through layer normalization—which is itself a reduction over the hidden dimension. Layer norm amplifies small differences because it divides by a variance term computed from the same values. A tiny shift in the GEMM output becomes a slightly different variance, which becomes a slightly different normalization, which becomes a slightly different input to the next layer's attention. You can observe this directly via cuBLASLt's algorithm preference reporting: # cuBLASLt algorithm preference histogram (conceptual) # Track per-step which algorithm ID was selected for the primary GEMM # # Healthy (stable shapes): # algo_id=42 (WMMA_TENSOR_OP_HMMA_16816) → 99.2% of steps # algo_id=17 (SIMT_FP16_SPLITK) → 0.8% of steps # # Under shape churn (continuous batching, mixed lengths): # algo_id=42 (WMMA_TENSOR_OP_HMMA_16816) → 61.3% of steps # algo_id=17 (SIMT_FP16_SPLITK) → 22.1% of steps # algo_id=31 (WMMA_TENSOR_OP_PAD16) → 16.6% of steps # # When algo_id distribution churns, your reduction tree is churning. # When your reduction tree churns, your last bits are churning. # When your last bits churn under thin margins, your tokens can flip. That histogram is the smoking gun. When you see algorithm preference distribution widening under load, you're watching the tensor cores get destabilized by shape churn. The fix isn't "use bigger batches." The fix is to understand that continuous batching creates a shape distribution, not a fixed shape, and that shape distribution maps directly to a tactic distribution, which maps directly to a ulp distribution. NCCL causal backpressure chains across TP+DP pods Now scale this to the fabric. Take an 8×TP + 4×DP pod: 32 GPUs total, where every token step requires AllReduce across the 8-way TP group, and gradient synchronization (or KV redistribution in some architectures) across the 4-way DP group. Here's the causal backpressure chain I've traced in production, laid out as a timeline: Rank 5 (of 8 TP ranks) hits a quant/dequant stall. Its KV blocks are fragmented, workspace is tight, and the runtime forces a dequant cycle mid-attention. That adds ~1.2ms to this rank's compute. AllReduce stalls on Rank 5. The other 7 ranks complete their portion and issue their NCCL send. Rank 5 hasn't arrived yet. NCCL's ring/tree protocol can't progress past this rank. Effective t_sync inflates by 2× compared to the no-straggler baseline. P2P retransmit triggers. Under some fabric topologies and congestion states, the delayed arrival from Rank 5 can cause NCCL to hit internal retry logic on the NVLink or InfiniBand path. This is not a "network error"—it's the transport protocol managing flow control under backpressure. But it adds latency jitter that is invisible unless you're tracing at the NCCL bootstrap level. vLLM scheduler reacts to the stretched step. The scheduler sees that step t took 2× longer than expected. Under its latency-aware admission control, it drops batch size from 32 → 12 to protect SLO. Smaller batch means different shapes. Different shapes mean different tactics. The plan changes. The batch size drop propagates. With batch size at 12, queued requests wait longer. Queue pressure builds. When the scheduler recovers and re-admits, the burst creates shape churn. Shape churn destabilizes tensor core fragment utilization. The system is now in a different execution regime—triggered by one rank's memory fragmentation. That is a causal backpressure chain. Not a latency spike. Not a network blip. A causally connected sequence where a microarchitectural event on one device reshapes the execution plan across the entire pod. To trace this, you need NCCL bootstrap traces with NVTX domain annotations: # NCCL tracing with NVTX domains for causal analysis # # Environment setup for trace collection: # NCCL_DEBUG=INFO # NCCL_DEBUG_SUBSYS=INIT,COLL,P2P # NSYS_NVTX_DOMAINS=nccl,cuda,cublas # # In Nsight Systems, correlate: # 1. Per-rank kernel duration (cuda domain) — identify the straggler # 2. NCCL collective start/end (nccl domain) — measure t_sync inflation # 3. P2P transport events (nccl/P2P) — detect retransmit/backpressure # 4. Scheduler batch decisions (application NVTX) — see batch size reaction # # The causal signal: when rank N's kernel duration spike aligns with # NCCL collective inflation across all ranks, followed by batch size # reduction in the scheduler, you have a causal backpressure chain. # # Regex for filtering straggler events in nsys export: # grep -E "ncclAllReduce.*duration_us > (2 * median_duration)" trace.sqlite # → correlate timestamp with scheduler batch_size change events This is the telemetry that separates "we think there was network jitter" from "Rank 5's dequant stall caused a 2× collective inflation that forced the scheduler to halve batch size, which shifted the shape class into a non-WMMA tactic for the next 47 steps." The first is a guess. The second is a causal explanation. And in an incident review at scale, only the second one survives. ISR + checkpoint overlap pathology: memory evaporation under replication pressure This is the deepest pathology in this article, and it almost never surfaces below 512 sequences per second. Large-scale inference deployments use incremental state replication (ISR) for fault tolerance: rather than checkpointing the entire model state, you replicate KV cache deltas and scheduler state to a standby node incrementally, so failover is fast. Separately, many systems run async checkpointing for recovery: periodic snapshots of model and optimizer state written to persistent storage, overlapped with inference to avoid blocking the decode loop. Under normal load, these two systems coexist peacefully. ISR replicates small deltas. Checkpointing writes in the background. Memory headroom is sufficient for both. Under paging pressure—the exact regime we've been discussing throughout this article—they collide. Here's the pathological interaction: The system is under VRAM pressure. KV blocks are being paged (allocated, evicted, re-allocated) at high frequency. Memory headroom is thin. ISR kicks in. It needs to replicate recent KV deltas to the standby. To do this, it must pin certain KV blocks in memory while it serializes and transmits them. Async checkpointing overlaps. The checkpoint writer is also holding references to memory regions it's snapshotting. Under normal conditions, this is fine—there's enough headroom. Under paging pressure, the checkpoint's memory holds compete with ISR's memory holds. Memory evaporation. The combined pinning from ISR + checkpointing temporarily removes KV blocks from the pool available to the decode loop. The pager sees available blocks drop. It may be forced to evict active KV blocks—blocks that are needed for in-flight sequences—to make room. Evicted blocks must be recomputed. When a sequence's KV is evicted mid-collective (during an AllReduce, for example), the rank that lost its KV must recompute it. That recompute makes this rank the straggler. And we already know what stragglers do to the collective timeline. The straggler triggers the full backpressure chain. Collective stall → batch size reduction → shape churn → tactic churn → output drift. All caused by a fault-tolerance mechanism designed to keep you safe. ISR pins KV deltas for replication while async checkpointing pins regions for snapshotting. Under paging pressure, the combined pinning shrinks the decode-available KV pool, forces evictions and recompute, creates stragglers, and cascades into collective stalls → batch reduction → shape/tactic churn → p99 output drift. I call this memory evaporation because from the decode loop's perspective, VRAM that was available simply vanishes for a window of time. The blocks are still physically present—they're held by ISR and the checkpointer, but they're not available to the runtime. The effect is identical to a sudden drop in free VRAM, and the runtime reacts accordingly: it enters a pressured regime. This is why the pathology rarely surfaces below 512 seq/s. At lower throughput, there's enough headroom that ISR and checkpointing never compete meaningfully with the decode loop's memory needs. At high throughput under sustained load, the margins collapse, and the three systems—decode, ISR, checkpoint—start fighting over the same memory. The fix is not "turn off ISR." The fix is to coordinate memory budgets across these three subsystems and to treat ISR and checkpointing as memory consumers that participate in the regime calculation. If your regime function doesn't account for replication and checkpoint holds, it's underestimating pressure, and your system will surprise you at exactly the scale where fault tolerance matters most. # extended regime function accounting for replication and checkpoint pressure def regime_extended(vram_free_mb, paging_on, isolation_strict, queue_p95_ms, isr_pinned_mb, ckpt_pinned_mb, kv_pool_total_mb): effective_free = vram_free_mb - isr_pinned_mb - ckpt_pinned_mb effective_ratio = effective_free / kv_pool_total_mb if kv_pool_total_mb > 0 else 1.0 if isolation_strict: return "isolation_strict" if effective_ratio < 0.05: return "memory_evaporation" # ISR+ckpt collision if paging_on: return "paging" if effective_free < 1024: return "memory_pressured" if queue_p95_ms > 50: return "queue_degraded" return "normal" That "memory_evaporation" regime is the one you never see at idle. It only appears when throughput is high enough that ISR frequency, checkpoint frequency, and decode memory demand all peak simultaneously. And when it appears, it doesn't show up as an OOM. It shows up as a straggler, which shows up as a collective stall, which shows up as a batch size drop, which shows up as a shape change, which shows up as output drift at p99. That's the full causal chain from fault tolerance to token flip. The chip-architect handoff These four pathologies, wavefront divergence, tensor core fragmentation, NCCL backpressure, and ISR collision are what elevate from principal-level operational insight to chip-architect-level systems thinking. They share a common structure: A microarchitectural or infrastructure event occurs that is invisible at the API layer. The event changes the timeline or the memory topology, not the "inputs." The changed timeline or topology feeds back into scheduling, shaping, or tactic selection. The feedback loop produces a different executed plan. The different plan produces a different result that is correct by contract but different by observation. If you're instrumenting at this depth, you're not debugging anymore. You're operating a system where the observability itself is part of the architecture. And if you're carrying the thesis of this article to its logical conclusion: the executed plan is not just a function of the GPU state. It's a function of the warp state, the fragment state, the fabric state, and the replication state—all coupled through continuous batching at token cadence. Security is not a layer, it changes execution Now let’s go deep, because this is where a lot of principal level reviews go wrong. Teams talk about security as confidentiality and correctness as something separate. In multi tenant inference, they couple. IOMMU based GPU isolation and DMA remapping Microsoft documents IOMMU based GPU isolation as a technique to manage how GPUs access system memory, improving security and stability: Microsoft also documents IOMMU DMA remapping, describing how GPUs access memory through logical addresses that are no longer mapped one to one, enabling logically contiguous address ranges through translation: This matters for two reasons. First, it is a real hardware enforced boundary, not a policy checkbox. Second, boundaries introduce overhead and constraints. Constraints change what is allowed. Allowed execution choices shape the plan space. Confidential computing on H100 NVIDIA states that H100 is the first GPU to introduce support for confidential computing and that it can be used in virtualized environments with VMs or Kubernetes based deployments. Azure has also published general availability of confidential VMs with H100, which is the practical deployment side of this posture: Now the key architectural point. When you turn on stronger isolation, you often restrict sharing. You restrict cross tenant microbatching. You add attestation requirements. You change how memory is mapped and protected. That can reduce throughput. Reduced throughput moves you closer to regime boundaries. When the system crosses a regime boundary, the executed plan changes. Security posture becomes an SLO dimension. If you do not test it, you do not know what system you are running. GPU cache side channels, why sharing is not a theoretical risk There is published research that treats GPU caches as a leakage surface. The USENIX Security 2024 paper Invalidate plus Compare presents a timer free GPU cache attack primitive. I will not provide attack recipes. You do not need them to understand the conclusion. If your threat model includes untrusted co tenants, shared microarchitectural resources matter. If you respond by increasing isolation, your execution constraints change. That changes performance and can change the execution regimes your serving stack enters. Security and runtime behavior are coupled. State collapse, the phase transition that looks like model instability If you don’t know what state collapse is, imagine a highway that looks perfectly calm at 2 a.m. Every lane is open. Every car keeps its distance. Your ETA is stable. You run the same route ten times and you get the same arrival time. Then 8:30 a.m. hits. Nothing “broke” in the highway. The asphalt is the same. The speed limit is the same. The cars are the same. But the system crosses a density threshold. One small brake tap becomes a shockwave. Lanes start interacting. Merges become bottlenecks. A single slow truck creates a queue that ripples backwards. Suddenly your ETA isn’t a property of your car anymore. It’s a property of the traffic regime. That is state collapse in production inference. At low load, the system behaves stable. At high load, output drift appears. And teams mislabel it as “model instability,” or “LLM randomness,” or “temperature drift.” Most of the time, it is none of that. It is a phase transition in the runtime. You didn’t change weights. You crossed a regime boundary. What collapses, exactly State collapse is not “everything gets slower.” It is when the control plane loses the degrees of freedom it was using to keep execution consistent. Under low load, the runtime has slack: enough VRAM headroom to keep preferred tactics feasible enough cache residency to keep step times predictable enough scheduling flexibility to keep microbatch composition stable enough workspace contiguity to avoid algorithm fallbacks enough fabric stability (in multi-node TP) to keep step cadence tight Under high load, that slack disappears. The runtime stops being a “fast executor” and becomes a “survival scheduler.” And once it crosses that boundary, it starts making different decisions that are all valid, all correct by contract, and all capable of shifting outputs. This is why it feels like instability: the model hasn’t changed, but the executed plan has. Why this shows up as output drift, not just latency drift Because decoding is a branching process. A small numerical difference that does nothing in a benchmark can flip a token if the margin is thin. One flip changes the context. The context changes the next logits. Now you’re on a different path. So the runtime doesn’t need to be “wrong” to produce different text. It just needs to execute a different legal plan under a different legal regime. That is the whole thesis of this article, condensed into one sentence: Weights are static. Behavior is a property of the executed plan. The executed plan is a function of state. The common triggers that push systems into collapse You can treat these as the usual “threshold crossings” that shrink the feasible plan space: Memory headroom shrinks → feasible tactic set shrinks Preferred kernels often require workspace. When headroom or contiguity drops, tactics become illegal and the engine selects other tactics. Cache residency collapses → stalls rise → step timing drifts L2 hit rate drops, HBM traffic rises, and decode steps stretch. In continuous batching, stretched steps reshape the next batch. Continuous batching shifts the mix and shapes Under load, microbatch membership changes at token cadence. Shape class changes are not cosmetic; they change kernel feasibility. Framework and engine algorithm selection changes depending on settings Autotuning, benchmarking, and backend heuristics mean the “same op” can legally choose different algorithms. Under pressure, the best choice can become infeasible. CUDA execution permits ordering freedom and floating point order sensitivity remains true Parallel staging and legal reordering can shift last bits. Under thin margins, last bits can become different tokens. Nothing here requires a bug. This is what “execution under constraint” looks like. The incident question that stops the hand-waving If you want a more honest incident question, use this: Which execution regime ran, and what constraints pushed us into it? Not “was the prompt the same.” Not “were the weights the same.” Not “did we set temperature to zero.” Regime first. Because state collapse is not a mystery. It’s a threshold. And once you learn to name the threshold, you can instrument it, test it, and stop being surprised by it at p95 and p99. A reproducibility protocol that works for principals, not demos Logging prompts is not reproducibility. It is wishful thinking. If you want to be able to defend behavior, you need to reconstruct the execution state. Log the execution contract Per request, log: effective input length after shaping truncation boundary and reason decode configuration actually applied admission time, queue time, GPU time per step batch fingerprint or at minimum batch identity and shape class memory headroom watermark and whether you were in a pressured allocation regime engine precision mode settings and any fallback relevant flags cuDNN benchmark and deterministic settings if relevant isolation posture, including whether cross tenant batching is permitted Track margins early Track top two logit margins for early steps. Use it as a stability budget. If the margin collapses under a certain prompt family, treat that as a risk surface. Not every prompt is equally stable. Test under regimes, not at idle Do not run determinism tests at idle and call it solved. Test under: sustained concurrency mixed sequence lengths continuous batching realistic memory pressure real isolation posture If you do not do this, you are validating a different system than the one you ship. vLLM’s paper exists precisely because these conditions define the serving problem. Closing If you want production LLM behavior to be explainable, stop treating the model as the whole system. Weights are static. Executed math is selected under constraint. Behavior lives in the gap. You did not deploy weights. You deployed a physics constrained runtime that contains weights. And that runtime is allowed to change the executed plan, because floating point order matters, CUDA scheduling freedom is part of the contract, engines can choose precision pathways, and serving stacks intentionally reshape batching and memory. Acknowledgments While this article dives into the hidden memory mechanics that shape LLM behavior under load, I’m grateful it was peer-reviewed and challenged before publishing. A special thanks for Hammad Atta and Abhilekh Verma for peer-reviewing this piece and challenging it from a security-and-systems angle. If this article resonated, it’s likely because it describes a reality many teams encounter only after an incident: production LLM behavior is a property of the executed plan, and the executed plan is a function of state. If you’re running production inference at scale and observing behavior shifts under load—especially in tail-latency regimes, I’m happy to connect on LinkedIn. I’m open to substantive technical discussion. Thank you for reading. I hope this helps you surface the hidden variables in serving and turn them into telemetry, controls, and repeatable postmortem evidence. And if you’re seeing similar regime transitions or plan churn in your own deployments, I’d be interested to hear how it presents in your stack. — Hazem Ali Microsoft AI MVP, Distinguished AI & ML Engineer / Architect
Looking for advice on collaborating with complementary Microsoft partners
a { text-decoration: none; color: #464feb; } tr th, tr td { border: 1px solid #e6e6e6; } tr th { background-color: #f5f5f5; } Hi everyone 👋 My name is Martin Rojze. I’m focused on the Microsoft data platform, with a specialization in Microsoft Fabric and Power BI. My work is centered on helping organizations design, implement, and scale modern analytics and reporting solutions on Azure, with a strong emphasis on real world business outcomes rather than just dashboards. As demand for end to end solutions continues to grow, I’m looking to deepen collaboration with complementary Microsoft partners, for example partners who focus on Dynamics 365 or Business Central Data engineering, data science, or AI App development including Power Apps, custom apps, or ISVs Security, governance, or change management I’d really appreciate advice from partners who have successfully built co sell or referral relationships, specifically What has worked and what has not when partnering with other Microsoft partners How you structure collaboration so it’s mutually beneficial and scalable Tips on aligning around go to market, co selling, or delivery without stepping on each other’s toes If you’re a partner interested in collaborating around Fabric and Power BI led analytics engagements, or if you’re willing to share lessons learned, I’d love to connect and learn from your experience. Thanks in advance and looking forward to the discussion. MartinLogic Apps Aviators Newsletter - March 2026
In this issue: Ace Aviator of the Month News from our product group News from our community Ace Aviator of the Month March 2026's Ace Aviator: Lilan Sameera What's your role and title? What are your responsibilities? I’m a Senior Consultant at Adaptiv, where I design, build, and support integration solutions across cloud and enterprise systems, translating business requirements into reliable, scalable, and maintainable solutions. I work with Azure Logic Apps, Azure Functions, Azure Service Bus, Azure API Management, Azure Storage, Azure Key Vault, and Azure SQL. Can you give us some insights into your day-to-day activities? Most of my work focuses on designing and delivering reliable, maintainable integration solutions. I spend my time shaping workflows in Logic Apps, deciding how systems should connect, handling errors, and making sure solutions are safe and effective. On a typical day, I might be: - Designing or reviewing integration workflows and message flows - Investigating tricky issues - Working with teams to simplify complex processes - Making decisions about patterns, performance, and long-term maintainability A big part of what I do is thinking ahead, anticipating where things could go wrong, and building solutions that are easy to support and extend. The culture at Adaptiv encourages this approach and makes knowledge sharing across teams easy. What motivates and inspires you to be an active member of the Aviators/Microsoft community? The Microsoft and Logic Apps communities are incredibly generous with knowledge. I’ve learned so much from blogs, GitHub repos, and forum posts. Being part of the Aviators community is my way of giving back, sharing real-world experiences, lessons learned, and practical solutions. Adaptiv encourages people to engage with the community, which makes it easier to contribute and stay involved. Looking back, what advice do you wish you had been given earlier? Don’t wait until you feel like you “know everything” to start building or sharing. You learn the most by doing, breaking things, fixing them, and asking questions. Focus on understanding concepts, not simply tools. Technologies change, fundamentals don’t. Communication matters as well. Being able to explain why something works is just as important as making it work. What has helped you grow professionally? Working on real-world, high-impact projects has been key. Being exposed to different systems, integration patterns, and production challenges has taught me more than any textbook. Supportive teammates, constructive feedback, and a culture that encourages learning and ownership have also been key in my growth. If you had a magic wand that could create a feature in Logic Apps, what would it be? I would love a first-class, visual way to version and diff Logic Apps workflows, like how code changes are tracked in Git. It would make reviews, troubleshooting, and collaboration much easier, notably in complex enterprise integrations, and help teams work more confidently. News from our product group New Azure API management service limits Azure API Management announced updated service limits across classic and v2 tiers to ensure predictable performance on shared infrastructure. The post details new limits for key resources such as API operations, tags, products, subscriptions, and users, along with a rollout schedule: Consumption/Developer/Basic (including v2) from March 15, Standard/Standard v2 from April 15, and Premium/Premium v2 from May 15, 2026. Existing classic services are grandfathered at 10% above observed usage at the time limits take effect. Guidance is provided on managing within limits, evaluating impact, and requesting increases (priority for Standard/Standard v2 and Premium/Premium v2). How to Access a Shared OneDrive Folder in Azure Logic Apps Logic Apps can work with files in a OneDrive folder shared by a colleague, but the OneDrive for Business “List files in folder” action doesn’t show shared folders because it enumerates only the signed‑in user’s drive. The article explains two supported approaches: (1) call Microsoft Graph using HTTP with Microsoft Entra ID (delegated permissions), or (2) use Graph Explorer to discover the shared folder’s driveId and folderId, then manually configure the action with {driveId}:{folderId}. A troubleshooting section shows how to extract these identifiers from browser network traces when Graph Explorer results are incomplete. Stop Writing Plumbing! Use the New Logic Apps MCP Server Wizard A new configuration experience in Logic Apps Standard (Preview) turns an existing logic app into an MCP server with a guided, in‑portal workflow. The wizard centralizes setup for authentication, API keys, server creation, and tool exposure, letting teams convert connectors and workflows into discoverable MCP tools that agents can call. You can generate tools from new connectors or register existing HTTP‑based workflows, choose API key or OAuth (EasyAuth) authentication, and test from agent platforms such as VS Code, Copilot Studio, and Foundry. The post also notes prerequisites and a known OAuth issue mitigated by reapplying EasyAuth settings. Logic Apps Agentic Workflows with SAP - Part 2: AI Agents Part 2 focuses on the AI portion of an SAP–Logic Apps integration. A Logic Apps validation agent retrieves business rules from SharePoint and produces structured outputs—an HTML summary, a CSV of invalid order IDs, and an “invalid rows” CSV—that directly drive downstream actions: email notifications, optional persistence of failed rows as custom IDocs, and filtering before a separate analysis step returns results to SAP. The post explains the agent loop design, tool boundaries (“Get validation rules,” “Get CSV payload,” “Summarize review”), and a two‑model pattern (validation vs. analysis) to keep AI outputs deterministic and workflow‑friendly. Logic Apps Agentic Workflows with SAP - Part 1: Infrastructure Part 1 establishes the infrastructure and contracts for a Logic Apps + SAP pattern that keeps integrations deterministic. A source workflow sends CSV data to SAP, while destination workflows handle validation and downstream processing. The post covers SAP connectivity (RFC/IDoc), the SAP‑side wrapper function, and the core contract elements—IT_CSV for input lines, ANALYSIS for results, EXCEPTIONMSG for human‑readable status, and RETURN (BAPIRET2) for structured success/error. It also details data shaping, error propagation, and email notification paths, with code snippets and diagrams to clarify gateway settings, namespace‑robust XPath extraction, and end‑to‑end flow control. Azure API Management - Unified AI Gateway Design Pattern This customer‑implemented pattern from Uniper uses Azure API Management as a unified AI gateway to normalize requests, enforce authentication and governance, and dynamically route traffic across multiple AI providers and models. Key elements include a single wildcard API, unified auth (API keys/JWT plus managed identity to backends), policy‑based path construction and model‑aware routing, circuit breakers with regional load balancing, token limits and metrics, and centralized logging. Reported outcomes include an 85% reduction in API definitions, faster feature availability, and 99.99% service availability. A GitHub sample shows how to implement the policy‑driven pipeline with modular policy fragments. A BizTalk Migration Tool: From Orchestrations to Logic Apps Workflows The BizTalk Migration Starter is an open‑source toolkit for modernizing BizTalk Server solutions to Azure Logic Apps. It includes tools to convert BizTalk maps (.btm) to Logic Apps Mapping Language (.lml), transform orchestrations (.odx) into Logic Apps workflow JSON, map pipelines to Logic Apps processing patterns, and expose migration tools via an MCP server for AI‑assisted workflows. The post outlines capabilities, core components, and command‑line usage, plus caveats (e.g., scripting functoids may require redesign). A demo video and GitHub repo links are provided for getting started, testing, and extending connector mappings and migration reports. Azure Arc Jumpstart Template for Hybrid Logic Apps Deployment A new Azure Arc Jumpstart “drop” provisions a complete hybrid environment for Logic Apps Standard on an Arc‑enabled AKS cluster with a single command. The deployment script sets up AKS, Arc for Kubernetes, the ACA extension, a custom location and Connected Environment, Azure SQL for runtime storage, an Azure Storage account for SMB artifacts, and a hybrid Logic Apps resource. After deployment, test commands verify each stage. The post links to prerequisites, quick‑start steps, a demo video, and references on hybrid deployment requirements. It invites community feedback and contributions via the associated GitHub repository. News from our community Pro-Code Enterprise AI-Agents using MCP for Low-Code Integration Video by Sebastian Meyer This video demonstrates how Model Context Protocol (MCP) can bridge pro-code and low-code integration by combining Microsoft Agent Framework with Azure Logic Apps. It shows how an autonomous AI agent can be wired into enterprise workflows, using MCP as the glue to connect to systems and trigger actions through Logic Apps. Viewers see how this approach reduces friction between traditional development and low-code automation while enabling consistent orchestration across services. The result is a practical pattern for extending enterprise automation with agent capabilities, improving flexibility without sacrificing control. Logic Apps: Autonomous agent loops - a practical solution for application registration secrets expiration (part 1) Post by Şahin Özdemir Şahin Özdemir describes how a single expired client secret disrupted an integration platform and how Logic Apps autonomous agent loops can prevent recurrence. The solution uses an AI-backed agent loop to call Microsoft Graph, list app registrations, detect secrets expiring within three weeks, and notify stakeholders via email using the Office 365 connector. Prerequisites include a Logic App with a managed identity and an AI model (e.g., via Microsoft Foundry). Clear agent instructions and tool context are emphasized to ensure consistent behavior. The result is a low-effort operational guardrail that replaces complex control-flow logic. From Low-Code to Full Power: When Power Platform Needs Azure with Sofia Platas Video by Ahmed Bayoumy & Robin Wilde Robin Wilde hosts Sofia Platas to explore when Power Platform solutions should extend into Azure. The conversation focuses on adopting an engineering mindset beyond low-code constraints—recognizing when workloads need Azure capabilities for scale, integration, or specialized services. It highlights moving from CRM and Power Platform into Azure and AI, and how pushing boundaries accelerates growth. The episode emphasizes practical decision-making over rigid labels, encouraging builders to reach for Azure when required while retaining the speed of low-code. It’s an insightful discussion about balancing agility with the robustness of cloud-native architecture. Cut Logic Apps Standard Costs by 70% in Dev & POC Azure Environments Post by Daniel Jonathan This article explains a practical cost-saving pattern for Logic Apps Standard in non‑production environments. Because Standard runs on an App Service Plan billed continuously, the author recommends deploying compute only during working hours and tearing it down afterward while retaining the Storage Account. Run history persists in storage, so redeployments reconnect seamlessly. Scripts automate deploy/teardown, with guidance on caveats: avoid removing compute during active runs, recurrence triggers won’t “catch up,” and production should stay always‑on. The post compares Standard versus Consumption and shows how this approach typically yields around 70% savings. Friday Fact: You can reference App Settings inside your Logic Apps Workflows Post by Sandro Pereira Sandro Pereira highlights a simple technique to externalize configuration in Logic Apps Standard by using the appsetting('Key') expression directly in workflow actions. The approach allows storing connection details, flags, and endpoints in App Settings or local.settings.json rather than hardcoding values, improving maintainability and environment portability. He notes the expression may not appear in the editor’s suggestion list but still works when added manually. The post includes a concise “one‑minute brief” and reminders to ensure the keys exist in the chosen configuration source, plus a short video for those who prefer a quick walkthrough. LogicAppWorkbook: Azure Monitor Workbook for Logic Apps Standard (App Insights v1) Post by sujith reddy komma This open-source Azure Monitor workbook provides a focused dashboard for Logic Apps Standard using Application Insights v1 telemetry. It organizes monitoring into Overview and Failures tabs, surfacing KPIs, status distribution, execution trends, and detailed failure grids. The repository includes KQL queries (Queries.md), screenshots, and clear import steps for Azure Workbooks. Notably, it targets the v1 telemetry schema (traces table, FlowRunLastJob) and isn’t compatible with newer v2 telemetry without query adjustments. It’s a useful starting point for teams wanting quick visibility into run health and trends without building dashboards from scratch. Azure Logic Apps - Choosing Between Consumption and Standard Models Post by Manish K. This post shares a primer that compares Logic Apps Consumption and Standard models to help teams choose the right hosting approach. It outlines Standard’s single‑tenant isolation, VNET integration, and better fit for long‑running or high‑throughput workloads, versus Consumption’s multi‑tenant, pay‑per‑action model ideal for short, variable workloads. It highlights migration considerations, limitations, and when each model is cost‑effective. The takeaway: align architecture, networking, and workload patterns to the model’s strengths to avoid surprises in performance, security, and pricing as solutions scale. Logic Apps standard monitoring dashboard – Fix ‘Runs’ tab Post by Integration.team Integration.team describes a fix for Logic Apps Standard where the Application Insights “Runs” tab shows a misconfiguration error and no history. The solution has two parts: ensure host.json sets ApplicationInsights telemetry to v2, and add a hidden tag on the Logic App that links it to the App Insights resource. They provide Bicep snippets for automated deployments and a portal-based alternative during initial creation. After applying both steps, run history populates correctly, restoring visibility in the monitoring dashboard and making troubleshooting more reliable. Using MCP Servers with Azure Logic App Agent Loops Post by Stephen W Thomas Stephen W Thomas explains how exposing Logic Apps as MCP servers simplifies agent loop designs. By moving inline tool logic out of the agent and into MCP-exposed endpoints, tools become reusable, easier to debug, and scoped to only what an agent needs. He discusses limiting accessible tools to control cost and execution time, and outlines a structure for organizing Logic Apps as discrete capabilities. The approach reduces agent complexity while improving maintainability and governance for AI-enabled workflows on Azure. Logic App Best Practices, Tips, and Tricks: #49 The Hidden 32-Character Naming Trap in Logic Apps Standard Post by Sandro Pereira Sandro Pereira explains a subtle but impactful pitfall in Logic Apps Standard tied to the Azure Functions runtime: the host ID is derived from only the first 32 characters of the Logic App name. When multiple Logic App Standard instances share a storage account and have identical leading characters, collisions can cause intermittent deployment and runtime failures. He recommends ensuring uniqueness within the first 32 characters or, in advanced cases, explicitly setting the host ID via AzureFunctionsWebHost__hostid. The article includes naming patterns and practical guidance to avoid hours of troubleshooting.
