Build and Optimize a Data Lakehouse for Unified Data Intelligence
Hello Folks! Welcome back to the ITOpsTalk Blog and the Microsoft Azure Infrastructure Summit 2026 series. In this session James Baker and Sai Runtham, both from the Azure Data Lake Storage product team, take us through what a modern Lakehouse actually is, how to design one on Azure, and then they roll up their sleeves and build one end to end. If you have been hearing “Lakehouse” thrown around in architecture reviews and were not 100% sure what it changes for you as an IT Pro, this one is for you. 📺 Watch the session: Why IT Pros Should Care You might be thinking, “I run infrastructure, not analytics.” Fair point. But here is the thing. The lakehouse is increasingly the platform your business will run BI dashboards, AI agents, and decision support systems on, and you are the one who has to keep the data safe, governed, and reachable. Here is what is in it for you: It is a platform conversation. James spends a big chunk of the session on horizontal platform capabilities (storage, catalog, identity, secrets, network, policy) versus vertical pipeline concerns. That is squarely an IT Pro problem. Data is the asset. Workspaces, query engines, and dashboards are transient. The data lives forever, and protecting it is on your plate. Governance is what stops your data lake from rotting into a data swamp. Scale is a virtuous cycle. More data drives more insight, which drives more data. Your platform cannot become the ceiling. AI agents are the new consumers. They do not just read dashboards, they query gold tables directly. Your network, identity, and access controls have to keep up. What is a data lakehouse A data lakehouse is exactly what it sounds like. You take the cheap, flexible, schema-light scale of a data lake, and you fuse it with the low-latency query performance, update semantics, and governance of a data warehouse. One copy of the data. One place to govern it. No more forking from the lake into a warehouse just to make BI tools happy. Quick contrast: Data lake. Big, cheap, flexible. No schema enforced on write. Historically prone to becoming a swamp. Data warehouse. Low-latency queries, updates, strong governance, structured. Hits a scale ceiling and costs more. Data lakehouse. Lake-scale storage, with a high-performance query layer and warehouse-grade governance sitting over the top. No data fork. The big shift is that the data does not move. Your BI dashboards, your AI agents, your serverless SQL queries, they all hit the same governed tables in the lake. That keeps lineage clean and your security model sane. Building it on Azure James and Sai are clear that the architecture is less a fixed diagram and more a list of platform capabilities you compose. Here is the shape of it on Azure. Storage layer (the asset). Azure Data Lake Storage Gen2 (ADLS) with hierarchical namespace turned on. That is non-negotiable for analytics workloads. It gives you atomic directory operations, POSIX-style ACLs, and the performance Delta Lake relies on. OneLake in Microsoft Fabric if you want a tenant-wide logical lake that is built on ADLS Gen2 and stores everything in open Delta Parquet by default. Table format and pipelines. Open table formats: Delta Lake (and Apache Iceberg as it converges) give you ACID transactions, time travel, schema evolution, and streaming on cheap object storage. Azure Databricks Lakeflow Declarative Pipelines with Autoloader for incremental ingestion of both batch and streaming sources straight into Delta tables. Autoloader handles new file discovery, schema inference, and evolution for you. The medallion architecture for stamping out repeatable pipelines: o Bronze. Raw, append-only landing zone. Source of truth. o Silver. Cleansed, deduplicated, conformed, enriched. o Gold. Business-ready, aggregated, performance-optimized for consumption. Governance and identity. Unity Catalog as the single source of truth for catalog, lineage, and fine-grained access control across bronze, silver, and gold. Entra ID for identity. Managed identities for compute. Key Vault for secrets. Network protection around the perimeter. The data is the crown jewel, so private endpoints, firewalls, and VNet-attached compute are baseline. Consumption layer. Power BI Direct Query against a serverless SQL warehouse on the gold tables. No data copies, governance flows through. AI agents like Databricks Genie pointed at gold tables. Natural-language questions, live lineage, no data movement. The demo that ties it together. Sai walked through a real pipeline: NYC TLC taxi trips enriched with NOAA weather and ESPN/MLB sports events, ingested by Autoloader into bronze, transformed through silver, aggregated into gold. A parallel streaming pipeline handled synthetic live events for a real-time demand view. Power BI dashboards hit gold via Direct Query. And Genie answered questions like “which zones are most sensitive to sport events” by mapping demand around Madison Square Garden, with the query and the chart generated for you. All against the same lakehouse, no data movement, full lineage. Optimizing for cost and performance This is where a lot of lakehouses go sideways. A few things from the session and from the official guidance worth pinning to your wall: Get hierarchical namespace right. It is the difference between atomic directory operations and “copy then delete,” which is slow and expensive at scale. Use storage tiers and lifecycle policies. Hot for working data, Cool or Cold for older partitions, Archive for compliance retention. Lifecycle rules on ADLS do this automatically. Partition and file-size matter. Lots of tiny files kill query performance. Use OPTIMIZE, Z-Order, or liquid clustering on Delta tables, and partition on the columns your queries actually filter on. Lean on vectorized reads. ADLS plus Delta plus modern query engines push a lot of work down to columnar Parquet, which keeps your compute bill in check. Use serverless SQL warehouses where it fits. Direct Query against a serverless endpoint scales compute to demand and lets you keep dashboards fresh without import refreshes. Observe data, not just systems. “Is Databricks up” is necessary but not sufficient. Watch data freshness, row counts, pipeline blockages, and SLAs on the data itself. Govern everything. A well-governed lakehouse drives trust, which drives use, which drives value. Skipping governance early always costs more later. Getting Started If you want to put hands on a keyboard this week: Spin up an Azure Storage account with hierarchical namespace enabled. That is your ADLS Gen2 foundation. Stand up an Azure Databricks workspace, enable Unity Catalog, and point it at your ADLS account. Create a Lakeflow Declarative Pipeline. Use Autoloader to ingest a sample dataset (the NYC taxi data is a classic starting point) into a bronze Delta table. Add silver and gold notebooks or pipelines that clean and aggregate the data. Connect Power BI to a serverless SQL warehouse on your gold tables with Direct Query. If you are a Fabric tenant, mirror or shortcut data into OneLake and try the same pattern there, no infra to manage. Read the Hitchhiker’s Guide to ADLS before you scale up. It will save you future you a lot of grief. Resources Introduction to Azure Data Lake Storage The Hitchhiker’s Guide to the Data Lake Microsoft OneLake documentation Azure Databricks documentation Delta Lake on Azure Databricks Design Delta Lake architecture and medallion patterns Implement medallion lakehouse architecture in Microsoft Fabric Watch the rest of the Summit This session is one stop on a big tour. The full Microsoft Azure Infrastructure Summit 2026 playlist covers everything from sovereign cloud and AKS networking to backup, storage, and AI-assisted operations. If your job touches Azure, there is something in here for you. Head over to the full playlist and binge what is useful: https://www.youtube.com/playlist?list=PLjt5SKzX1iI8con7FJDB56G6hHqxGm7ki Cheers! Pierre RomanWhat to Do When You Hit Capacity in Azure Databricks: Engage, Mitigate, Plan!
Microsoft's Cloud Architects: Paul Singh PaulSingh, Aladdin Alchalabi aalchalabi, Eduardo Dos Santos eduardomdossantos, Chris Walk cwalk, Peter Lo PeterLo, Tim Orentlikher tim_orentlikher, Ajmal Hossain ajmalhossain, Chris Haynes Chris_Haynes, and Rafia Aqil Rafia_Aqil Start Here: Engage Microsoft Capacity constraints in Azure Databricks are not an Azure Databricks product issue. Azure Databricks does not own or reserve compute, it dynamically provisions VMs from Azure when clusters are created or scaled. This means cluster creation, autoscaling, or job execution can stall when the underlying VM SKUs are constrained at the regional level. The fastest path to resolution is a structured conversation with your Microsoft account team, who can engage the Azure capacity intake process on your behalf. Create a Quota Support Ticket via Microsoft Support and bring the following to your account team with your Support Ticket Number. Each field maps directly to what capacity intake teams will ask for: missing fields slow the request. What to Prepare Before You Reach Out Your Account Team Field What Capacity Intake Needs Example Subscription IDs The exact Azure subscriptions that will host the workspaces and clusters 7ebee83d-7923-426c-8449-59fd4dff25ab Region(s) Primary region, plus any acceptable alternates East US 2 VM family / SKU Specific series and version requested Eadsv5, ESv4, DSv4, DSv2 Core count / new limit Total vCPU or core count per SKU 10,000 cores for Eadsv5 Workload characteristic CPU-bound vs. memory/shuffle-heavy vs. IO-heavy; batch vs. streaming vs. SQL “Memory-intensive ETL with large joins and shuffles” Scale and timing When you need it, ramp profile, peak vs. steady state “Need by month-end; ramp from 2,000 to 9,650 cores over Q3” Business context Business use case “Migration off AWS” What “Capacity” Really Means: A Layered Mental Model Before diving into fixes, it is important to understand what is actually happening behind the scenes. Capacity constraints can occur at three distinct layers, and solving them requires addressing each one. Layer 1: Azure Infrastructure This is the layer most teams underestimate. Capacity here is governed by: VM SKU availability in the region. D-series and E-series: the two most common Databricks worker families: have repeatedly hit capacity constraints across multiple Azure regions, causing cluster creation failures, autoscale stalls, and provisioning delays. Regional supply constraints, which are dynamic and shared across all Azure tenants. vCPU quotas and limits per subscription, which are separate from regional supply. Quota is your subscription’s limit to deploy resources (like a credit card limit); regional capacity is the underlying infrastructure available. Both must be sufficient. Layer 2: Azure Databricks Platform The Azure Databricks control plane has its own published ceilings that your architecture must proactively respect. Key limits from the official Azure Databricks resource limits documentation: Resource Limit Scope Jobs created per hour 10,000 Workspace Tasks running simultaneously 2,000 Workspace (Run Job and For Each parent tasks excluded) Parent tasks running simultaneously (Run Job / For Each) 750 Workspace SQL warehouses 1,000 Workspace Attached notebooks or execution contexts 145 Cluster Virtual machines 25,000 Per subscription per region Note: For limits marked as non-fixed in the official documentation, you can request an increase through your Azure Databricks account team. Reference: https://learn.microsoft.com/en-us/azure/databricks/resources/limits Layer 3: Workload (Spark Execution) Even when both lower layers cooperate, Spark’s own execution model can produce capacity-like symptoms: Parallelism and task distribution, which dictate how many cores a job can usefully consume. Memory pressure from joins, shuffles, and skewed keys. IO demand and caching behavior, including Delta cache effectiveness and Spark cache misuse. Understanding these layers is critical. Retries sometimes succeed because capacity is dynamic: as other workloads complete, nodes are released back to Azure and briefly become available. Recognizing When You’ve Hit Capacity Capacity issues rarely present as a single clean error. Instead, they appear as inconsistent behaviors: Clusters stuck in Pending state Autoscaling fails or never reaches the desired size Jobs intermittently fail to start Retry attempts sometimes succeed These inconsistencies occur because capacity is shared across Azure tenants and fluctuates throughout the day. Running workloads outside peak business hours in the impacted region’s time zone is one of the most effective short-term mitigations. Immediate Actions: How to Unblock Your Workloads When you are actively hitting capacity constraints, speed matters. Please reach out to your Microsoft Account team and try these mitigations that are ordered from quickest to most involved. Retry and Run During Off-Peak Hours Capacity availability changes throughout the day as workloads complete and release VMs. Running outside peak business hours for the impacted region significantly improves success rates. Switch VM SKU or Family If a specific VM SKU is constrained, switching to another can immediately unblock provisioning. Move within the same family (for example, DSv4 → DSv5) Or switch families entirely (for example, D-series → F-series or L-series) This is one of the most effective but often underused approaches. Also, Choosing the Right VM Family Most Databricks environments default to D-series (general purpose) and E-series (memory optimized). These are also the most heavily used and most capacity-constrained VM families. Consider alternatives based on your workload: VM Family Best For When to Use Trade-off D-series General workloads Default choice Often constrained in high-demand regions E-series Memory-heavy Spark jobs Joins, shuffles, analytics High demand; higher cost F-series CPU-intensive jobs Parsing, transformations Lower memory per core L-series IO-heavy workloads Delta caching, large datasets Higher cost; large local NVMe Practical decision framework: Memory-bound workloads (joins, shuffles): Move from E-series to L-series. Similar memory per core, plus large local NVMe for Delta caching. CPU-bound workloads: Move from D-series to F-series. Higher CPU performance at lower cost. IO-heavy or cache-sensitive workloads: L-series can significantly improve performance and reduce shuffle pressure. Designing a single VM family is one of the biggest production risks in Azure Databricks environments. Implement Regional Diversity in your Databricks workload As Azure capacity constraints are region- and SKU-specific, it is important to build architectural flexibility into your Databricks deployments. For critical or large-scale workloads, consider deploying multiple Databricks workspaces across different Azure regions to reduce dependency on any single region’s capacity. This approach enables: improved resilience to regional capacity constraints greater flexibility in workload placement Important: Multi-region deployment requires deliberate architecture, including deploying separate workspaces and replicating data and configurations across regions—it is not automatic. Why Adding More Nodes Is Not Always the Answer When jobs slow down, the instinct is to scale compute. With Spark, more nodes do not always solve the problem. Common workload issues that masquerade as capacity problems: Data skew Excessive shuffle operations Inefficient partitioning Overuse of UDFs In real workloads, shuffle operations can grow significantly larger than input data, placing heavy pressure on both compute and memory that more nodes cannot relieve. Smarter optimization strategies: Reduce shuffle through repartitioning and query optimization Enable Photon for faster execution Optimize Delta tables using Z-ordering and compaction Leverage caching strategically (not just Spark cache: use the Delta/disk cache) These optimizations can reduce your dependency on scarce VM capacity altogether. What to Do When Your Capacity Is Approved Once Azure approves your capacity request, retaining it requires active steps. Because Azure capacity is dynamic and shared, approved capacity is held only while compute remains actively deployed and running. This is especially important in highly constrained regions. Microsoft recommends the following: Configure an Instance Pool For workloads that cannot yet use serverless compute, configure an Azure Databricks Instance Pool with a minimum number of idle nodes aligned to your production requirements. An instance pool pre-allocates and maintains a set of idle, ready-to-use VM instances. When a cluster is created from the pool, it draws from these warm nodes: eliminating the need to request new VMs from the regional Azure capacity pool between job runs. Key behaviors: The pool holds a minimum number of nodes continuously, keeping them warm and immediately available. Clusters attached to the pool pull from warm nodes, avoiding re-acquisition from Azure between runs. No DBU charges apply while nodes are idle in the pool. Azure VM infrastructure costs do apply for all minimum idle instances. Size the pool conservatively: aligned to production need only: to balance capacity retention against ongoing cost. Important: Instance pools hold idle nodes on a best-effort basis. Periodic platform events can recycle pool nodes, briefly causing the pool to fall below its configured minimum idle count while Azure re-acquires replacement nodes. Pools significantly improve availability and startup latency, but they do not change the fact that the underlying VMs are still requested from Azure on demand. They are not a hard reservation. Reference: https://learn.microsoft.com/en-us/azure/databricks/compute/pools Designing for Resilience: Long-Term Best Practices To avoid repeated capacity issues, your architecture needs to evolve beyond reactive mitigations. Plan for Capacity Early Understand VM quotas and limits before you need them: not after a constraint occurs. Avoid designing a single SKU. Build flexibility into cluster configurations so you can switch families without re-engineering jobs. Standardize Compute Configurations Consistent, policy-driven environments make it easier to adapt when capacity constraints occur. Use Databricks Cluster Policies to constrain cluster creation to approved, available VM families: this prevents teams from inadvertently requesting constrained SKUs. Move Toward Serverless Where Possible Serverless compute abstracts capacity management away from the customer. As the Databricks platform expands serverless support, migrating eligible workloads is the most durable long-term strategy. Azure continues to expand infrastructure capacity, but there are no guaranteed timelines for relief in constrained regions. Note: If your workload supports serverless compute, Databricks recommends using serverless compute instead of pools or classic VM-backed clusters. Serverless removes dependency on specific VM SKUs and regional capacity: scaling is managed by the platform with significantly improved availability. Reference: https://learn.microsoft.com/en-us/azure/databricks/serverless-compute. For eligible workloads: including Databricks Jobs (automated workflows), Databricks SQL Warehouses, and Delta Live Tables: serverless compute eliminates VM SKU dependency entirely. Configuration guidance is available in the Azure Databricks deployment guide, Development Section, Step 9. Multi-Region Strategy for Critical Workloads For the most critical workloads, evaluate a multi-region deployment as part of your business's continuity planning. This is a significant architectural investment: see the FAQ for the full scope: but it is the only approach that provides true regional redundancy. Coordinate this with your Microsoft account team. Reference: Azure Databricks & Microsoft Fabric Disaster Recovery: The Complete Better‑Together Strategy for Cloud Architects Final Takeaways Capacity issues are infrastructure-level constraints, not Databricks product failures VM family selection is critical: do not rely solely on D-series and E-series Workload optimization can reduce dependency on scarce resources before requesting more capacity Serverless compute is Microsoft’s preferred long-term recommendation for eligible workloads Azure On-Demand Capacity Reservations provide guaranteed capacity for mission-critical scenarios: distinct from instance pools (best-effort) and Reserved Instances (billing discount only) Architectural flexibility: multi-SKU, multi-region awareness is your best defense against future constraints FAQ Why do retries work? Capacity in Azure regions is shared across all tenants and fluctuates throughout the day as workloads complete and release VMs. A retry succeeds when capacity temporarily frees up. Retrying during off-peak hours improves success rates significantly. Why does capacity fluctuate during the day? Capacity is a function of regional supply and concurrent demand. As workloads complete, nodes are released back to Azure. Peak business hours in the impacted region’s time zone tend to be the tightest windows. Why are instance pools not a hard reservation? Pools hold a minimum number of nodes on a best-effort basis. Periodic platform events recycle pool nodes, so a pool can briefly fall below its configured minimum idle count while Azure re-acquires replacement nodes. Setting minimum idle to 0 avoids paying for idle VMs at the cost of slower acquisition time. Pools significantly improve availability and startup latency but do not guarantee capacity at the Azure infrastructure level. Why does serverless behave differently from classic clusters? Serverless compute removes customer control over individual VM SKUs. Databricks manages the underlying capacity across a shared pool. SKU-swap and pool-based mitigations do not apply. Customer-side levers reduce to retry and off-peak scheduling. The trade-off is that serverless is the simplest and most reliable option when the workload supports it. Why is changing regions a last resort? Region changes require redeployment of the Azure Databricks workspace and migration of all dependent artifacts: jobs, clusters, libraries, networking (private endpoints, VNet injection), Unity Catalog assignments, identities, and source data. The destination region must be validated for the same SKU and zonal configuration. For these reasons, region change should always be coordinated with the Microsoft account team and attempted only after preferred mitigations have been exhausted. Why does VM family selection matter so much for capacity? Different VM families have different supply curves. D-series and E-series are the most requested Databricks worker families and the ones most frequently constrained. Choosing a SKU based on whether the workload is memory/shuffle-heavy, CPU-bound, or IO-heavy improves both performance and the probability that capacity is available. The capacity team often steers customers toward newer-generation alternatives when supply differs by generation version. What does the Microsoft account team actually do? They route the request into the Azure capacity intake process, advise alternate SKUs and regions, surface zonal vs. regional considerations, and provide forward visibility into known constraints. The customer’s job is to bring a complete, accurate workload profile so the account team can advocate effectively. It is also recommended to open an Azure Support ticket. This will save time later, as the capacity planning teams would like to track issues and requests via a support ticket. Once an Azure Support ticket is opened, the ticket number should be shared to the Microsoft Account Team, at a minimum to the Customer Success Account Manager (CSAM), if one is assigned to your organization.Streaming and Batch Data Architectures with Microsoft Fabric to Azure Databricks
Author's: Aladdin Alchalabi aalchalabi, Oscar Alvarado oscaralvarado and Rafia Aqil Rafia_Aqil Note: This article describes a solution idea. Your cloud architect can use this guidance to help visualize the major components for a typical implementation. Use this article as a starting point to design a well-architected solution that aligns with your workload’s specific requirements. As organizations adopt Microsoft Fabric as their unified analytics platform, it has become a leading path for ingesting both streaming and batch data into Azure Databricks. This article covers integration approaches -via Microsoft Fabric- and details the five Fabric-specific paths that connect OneLake/ADLS and Databricks for end-to-end data processing. Medallion Architecture The following data flow corresponds to the architecture diagram: Data is ingested through Microsoft Fabric (via Mirroring, RTI, or Data Factory) lands data into OneLake/ADLS. With the medallion pattern, consisting of Bronze, Silver, and Gold storage layers, organizations have flexible access and extendable data processing: Bronze – Raw data entry point. Data arrives in its source format and is converted to the open, transactional Delta Lake format. Silver – Optimized for BI and data science. ETL and stream processing tasks filter, clean, transform, join, and aggregate Bronze data into curated datasets using SQL, Python, R, or Scala. Gold – Enriched data ready for analytics and reporting. Analysts use Power BI, PySpark, SQL, or Excel for insights and queries. Fabric Integration Paths Note: This architecture establishes a complete loop-back between Microsoft Fabric and Azure Databricks, enabling Gold layer tables to be seamlessly mirrored back to Microsoft Fabric for dashboarding through Azure Databricks Mirroring. The following five paths connect Microsoft Fabric to Azure Databricks: Fabric Mirroring to OneLake – A low-cost, low-latency turnkey solution that creates a replica of data from operational sources (SQL Server, Azure Cosmos DB, Oracle) in OneLake. Handles the initial load and ongoing CDC changes automatically, keeping data continuously up to date. Fabric RTI to OneLake – Fabric Real-Time Intelligence ingests streaming event data into OneLake with sub-second latency, enabling real-time analytics on live event streams. Fabric Data Factory to OneLake – Orchestrates ingestion from diverse sources not covered by Mirroring (such as Sybase or REST APIs) and lands data in OneLake, ensuring complete source coverage. OneLake to Azure Databricks – Unity Catalog connections to OneLake, secured via Managed Identities from Microsoft Entra ID, allow Databricks to query OneLake data items as a native catalog without data duplication. Fabric Data Factory to Azure Databricks (direct) – Orchestrates ingestion from diverse sources directly into Azure Data Lake Storage (ADLS), where Azure Databricks picks up the data for medallion architecture processing. Design Considerations Area Updated guidance Direct RTI-to-Databricks integration There is still no broad GA direct integration where Fabric RTI and Databricks operate as one native real-time runtime. Integration should be positioned through open protocols, Event Hubs/Kafka-style patterns, OneLake, Delta, and federation. OneLake federation in Azure Databricks OneLake federation in Azure Databricks is now the key integration story. It allows Databricks Unity Catalog to query Fabric Lakehouse and Warehouse data in OneLake without copying it. Access is read-only and depends on Fabric tenant settings, workspace permissions, and Databricks Unity Catalog setup. RTI data availability to Databricks Data ingested through Fabric RTI can be made available to Databricks by landing or exposing the data into OneLake-backed items, especially Lakehouse/Warehouse patterns. Eventhouse data can be made available in OneLake in Delta format through OneLake availability, but Databricks OneLake federation should be validated against the specific Fabric item type and access path. Existing Databricks customers Existing Databricks customers do not need to abandon Databricks. They can use Fabric RTI as the event ingestion, real-time detection, operational alerting, and business action layer, while continuing to use Databricks for engineering, ML, advanced analytics, and Unity Catalog-governed access. Activator and business action Fabric Activator is the cleanest business-user action layer. It can monitor streaming events and trigger Teams messages, email, Power Automate flows, Fabric pipelines, notebooks, Spark jobs, Dataflows, UDFs, and other downstream actions. This is a strong differentiator because it lets business users act on events without waiting for batch analytics. Operations Agents Operations Agents are in preview and should be positioned carefully. They monitor real-time data from Eventhouse or ontology sources, surface insights, recommend actions, and can connect to Activator/Power Automate action paths. They are not simply a pre-ingestion decision engine before data lands anywhere; they work from configured Fabric knowledge/data sources. Before landing in Lakehouse For decisioning before Lakehouse persistence, use Eventstream processing and Activator rules on streams. For AI-assisted operational recommendations, use Operations Agents once the relevant data is available in Eventhouse or ontology. Requirement-Specific Notes Data Ingestion Microsoft Fabric Mirroring currently supports SQL Server, Azure Cosmos DB, and Oracle as source systems. For sources not yet supported by Mirroring—such as Sybase or REST APIs—use Fabric Data Factory pipelines to ensure full coverage across all data systems. Once data is in the landing zone with the correct format, Mirroring’s CDC replication starts automatically and manages the complexity of merging changes (updates, inserts, and deletes) into Delta tables, keeping data in Fabric continuously up to date. Learn more about open mirroring Storage Format and Time Travel OneLake supports Delta tables, enabling schema evolution and time travel across all data stored in the lakehouse. Learn more about OneLake and Delta tables Security Encryption at rest: OneLake automatically encrypts all data at rest using Microsoft-managed keys, compliant with FIPS 140-2 standards. Learn more Encryption in transit: All data in transit is encrypted using TLS 1.2 or higher, securing data movement between Fabric, OneLake, and Azure Databricks. Learn more Data Governance OneLake can be registered and scanned by Microsoft Purview, enabling cataloging of stored metadata and data quality profiling. This protects sensitive information, including PHI and PII, across ingestion and analytics workflows. Learn more about Purview with Fabric Lakehouse Operations and Monitoring Use the Fabric monitor hub to track pipeline health, Spark application performance, and ingestion job status across all Fabric workloads. Learn more about the Fabric monitor hub Scenario Details This architecture applies to any organization that needs to unify streaming and batch data at scale. Common characteristics include: Multiple operational data sources (databases, SaaS applications, event streams) A requirement to process both real-time and historical data in the same platform Governance and compliance requirements for sensitive data (PHI, PII, financial records) Analytics consumers spanning BI (Power BI), data science (Databricks notebooks), and ML workloads Potential Use Cases Healthcare and life sciences – PHI/PII protection via Purview; real-time patient telemetry + batch EHR analytics Financial services – Real-time fraud detection streams + batch regulatory reporting Retail and e-commerce – Streaming clickstream analytics + batch inventory and supply chain processing Energy and utilities – IoT sensor telemetry streaming + batch consumption analytics Next Steps Get started with Microsoft Fabric Mirroring Build an ETL pipeline with Lakeflow Declarative Pipelines Configure Unity Catalog with OneLake shortcuts Monitor Fabric pipelines with the Fabric monitor hubToken Limit Exceeded? What's Actually Going On and What to Do About It ?
Hi All, Based on some recent experience across the organisation with token limit issues, I wanted to put my thoughts down and actually dig into what's happening under the hood, rather than just chalking it up to "we need a bigger plan." If you work anywhere near the Microsoft ecosystem these days, you're probably touching more AI tools than you realize. Copilot in Word and Excel, GitHub Copilot while you code, Copilot Studio if you're building agents, maybe Security Copilot or Copilot for Sales depending on your role, and increasingly Azure AI Foundry if your team is building anything custom. I work across a good chunk of this stack day to day, and at some point, almost everyone runs into the same wall: "Token limit exceeded." "You've reached your usage limit." "Upgrade to continue." The first instinct is usually to assume you did something wrong wrote too much, uploaded too big a file, or just need a fatter subscription. Sometimes that's the actual story. But honestly, often, that error message is standing in for three completely different problems that all happen to look identical from the outside. One is about how much text a model can physically process at once. One is about your license or credits running dry. And one has nothing to do with size at all it's just about how fast you're sending requests. Once you know which of these three, you're dealing with, the fix becomes obvious. Until then, "upgrade your plan" feels like the only lever you've got even when it isn't. This post walks through what a token is, why Microsoft's various Copilots each handle this differently, and what habits genuinely cut down on these interruptions instead of just throwing money at the problem. Part 1: So What Is a Token, Really? A token isn't a word, and it isn't a character it's somewhere in between. It's the small chunk of text a model's tokenizer breaks your input into before it can do anything with it. Take a word like "unbelievable." A tokenizer might split it into three pieces something like "un," "believ," and "able." Short, everyday words usually come out as a single token. But code, technical jargon, acronyms, and non-English text tend to fragment into a lot more tokens than you'd guess just by looking at the word count. This is why every AI tool has a ceiling on how much it can handle in one go, and that ceiling isn't measured in words or characters it's measured in tokens. Your prompt, any documents or emails it pulls in as context, the back-and-forth history of your conversation, and the response itself all draw from the same pool. Once that pool runs dry, something has to give: the tool truncates, rejects the request outright, or quietly summarizes older context to make room. The part that trips people up: token count doesn't map cleanly to word count. A short, dense paragraph full of code or acronyms can eat up more tokens than a much longer plain-English message. Part 2: Three Different Limits, One Confusing Error Message This isn't always obvious upfront, even to a lot of admins managing these tools: "token limit exceeded" is really a stand-in phrase for three separate limits, and they don't behave the same way. This isn't unique to Microsoft either every major AI platform bundles these same three things behind similarly vague error messages. Microsoft's stack just makes a good case study because so many of us touch multiple pieces of it in the same week. The context window is the ceiling on how much text a specific model can process in a single request everything from your prompt to retrieved documents to chat history. This is tied to the model itself, not your subscription. Swap from one model to another inside the same tool, and this ceiling can move without you doing anything differently. Your license, credits, or feature allowance is a completely separate thing. This is what Microsoft 365 Copilot plans track through AI credits and feature limits, and it's what Copilot Studio measures through Copilot credits at the environment level. A single action summarizing an inbox, generating an agent response, running an analysis deducts from this pool regardless of how small your actual prompt felt. Run out, and you get blocked, even if you're nowhere near any context window limit. The rate limit is about speed, not size. Copilot Studio, for instance, enforces quotas measured in requests per minute or per hour to keep the system stable under load. Send messages too quickly, which happens easily with automations, flows, or bots, and you can get throttled even with a tiny prompt and plenty of credits left. The reason this matters: a plan upgrade only ever fixes the second one. If you're actually running into the model's context window or getting rate-limited, paying for a bigger license won't change anything, and that mismatch is exactly where most of the frustration comes from. Part 3: How This Plays Out Across the Microsoft AI Stack The Microsoft ecosystem isn't one AI tool wearing different outfits it's genuinely several different systems, each handling tokens and limits in its own way. Here's a tour of the ones people run into most. Microsoft 365 Copilot (the one living inside Word, Excel, Outlook, Teams) doesn't work off a single published token number the way a developer tool would. Instead, it dynamically pulls together your prompt, recent chat history, and relevant snippets retrieved from Microsoft Graph your files, emails, and messages and quietly summarizes or drops older material to stay within bounds. Where this usually breaks isn't the context window at all; it's the AI credit and feature-limit system running out, often without much warning until you're mid-task. GitHub Copilot Chat is more like a traditional developer tool. It has a fixed, published token window tied to whichever model you've selected, and that limit applies consistently whether you're in the browser, VS Code, or the CLI. The failure mode here is usually a long conversation or a big multi-file context quietly creeping past that ceiling. Copilot Studio, where a lot of custom agent-building happens, runs on Copilot credits per interaction, plus its own requests-per-minute and requests-per-hour quotas at the environment level. If you're grounding an agent in SharePoint content, there's also a separate file-size ceiling to watch content over a certain size can get silently excluded from generative answers depending on your tenant's licensing. Azure AI Foundry (recently renamed to Microsoft Foundry, in case you've seen both names floating around) is where this gets more directly in your control. If your team is building custom applications on top of Azure OpenAI or other models in the Foundry catalog, which now includes everything from GPT to Phi to Claude to Llama, you're working with explicit, published context windows per model, and you're billed per token rather than per credit. It's a different mental model entirely: less "you hit a wall," more "you're paying by the word, so design accordingly." Security Copilot, if your org uses it for threat analysis and incident response, runs on its own capacity model pooled compute units at the tenant level rather than a simple per-user cap. It's easy to assume this behaves like M365 Copilot license limits; it doesn't. Copilot for Sales, embedded in Outlook and Teams for CRM-connected work, and Copilot in Power BI, which now goes beyond generating summaries to actually helping build and refine semantic models, both draw from their own feature-specific allowances layered on top of whatever base Microsoft 365 or Power Platform license you're on. And then there's the multi-model wrinkle that trips up teams the most: because tools like Copilot Studio and GitHub Copilot let you choose between GPT-based models, Claude, and others, the exact same prompt can have a different effective context window and a different token cost purely based on which model handled it that day. This is a big, underrated reason behind the "it worked fine yesterday, why not now" complaint. Part 4: What Actually Helps ? Some of this is genuinely outside your control, but a fair amount isn't. If you're just using these tools day to day, the single biggest habit shift is not letting conversations run forever. Long threads in Copilot Chat or Copilot Studio keep accumulating history, and that history eats into the same budget as whatever you're asking right now. Starting fresh periodically costs you nothing and buys back a lot of headroom. Large documents are worth splitting up before you feed them in, especially for SharePoint-grounded agents, where oversized files can get quietly excluded rather than cleanly rejected you won't necessarily know it happened unless you're looking for it. And it's worth resisting the urge to default to the heaviest, most capable model for every single task. Lighter models are usually faster, cheaper, and often sit under a more generous limit than the flagship ones, and most everyday tasks genuinely don't need the biggest model available. Before you go asking IT for a license upgrade, it's worth a quick sanity check on which limit you actually hit. If it's a rate limit, waiting a minute and retrying usually solves it outright. If it's a context window problem, trimming your prompt or starting a new session fixes it. An upgrade only helps if you've genuinely run out of credits or feature allowance, and that's worth confirming before you file the request. If you're on the building side Copilot Studio agents, Foundry applications, anything with RAG-style grounding a couple of things pay off quickly. Keep an eye on credit or token consumption proactively rather than discovering it's gone when the agent goes down mid-conversation. Be deliberate about what goes into system prompts and orchestration instructions, since those draw from the same budget as the end user's actual message, often invisibly to whoever's chatting with the agent. And spend real time getting chunk size right for knowledge sources too large and you're burning budget on irrelevant context, too small and the agent loses the thread. Part 5: Quick Checklist Before You Escalate Is this actually a context window problem -prompt, history, and attachments too big for the model in use? Have you genuinely run out of credits or feature allowance on your plan? Could this be a rate limit -too many requests too fast, especially from a flow or automation? Did the underlying model change since last time, quietly shifting the effective window? For Studio or Foundry work, is this a tenant or environment-level limit rather than something tied to you personally? Closing Thoughts Tokenization is one of those things that stays completely invisible right up until it isn't. Across a stack as sprawling as Microsoft's M365 Copilot, GitHub Copilot, Copilot Studio, Foundry, Security Copilot, and everything layered on top "token limit exceeded" almost never means one single thing. It means you've hit one of three very different walls, and each one needs a different response. If your team builds or maintains any of these tools, this is genuinely worth putting in front of people early. Most of the "why did this break" tickets in this space aren't about tokens at all. They're about nobody knowing which limit actually got hit, or where in this increasingly large ecosystem it happened. I'm curious how this shows up for others has your team standardized on one model across these tools, or are you juggling several depending on the task? I'd love to hear what patterns you've run into. Cheers, and happy reading. - By Surya Vennapusa, MCTWelcome Back to AZ Update
Hello Folks! Welcome Back to AZ Update A few years ago, Antony Bartolo and I launched a simple idea called AZ Update. The goal was to provide a place where IT professionals could quickly understand what was changing in Azure, why it mattered, and what they should pay attention to next. The show became a weekly conversation focused on Azure news, infrastructure, operations, security, and the real-world impact of Microsoft's latest cloud updates. Today, Azure is moving faster than ever. Every week brings new services, platform capabilities, operational improvements, AI innovations, and architectural guidance. Keeping up is a full-time job. Most of us don't have time to read every blog post, release note, announcement, and documentation update. That's why I'm bringing AZ Update back. This time, as a weekly LinkedIn newsletter and this blog. To be completely transparent I am using an AI Agent to parse the update list for any in the last 7 days, filter for Infra/Ops content and research product docs and help with the draft. I do review content and write the post myself. Each edition will cut through the noise and focus on what matters most for cloud architects, platform engineers, infrastructure teams, SREs, security professionals, and IT operators. I'll share the Azure announcements worth your attention, explain why they're important, highlight practical implications, and point you to the resources that can help you go deeper. Just a concise weekly briefing from one ITPro to another. If your day-to-day involves building, operating, securing, or modernizing infrastructure in Azure, Azure Arc, AKS, hybrid environments, or the growing world of AI-powered operations, this newsletter is for you. Welcome to the next chapter of AZ Update. Here is week 1! This week’s Azure infrastructure updates bring practical operational gains for security, platform reliability, disaster recovery, and identity-driven access control. Here is a detailed ITPro breakdown with implementation guidance you can use in production planning. Update #1 - Generally Available: Network Security Perimeter support for Azure Event Hubs Update #2 - Generally Available: Confidential Computing support for Azure Event Hubs Dedicated Update #3 - Generally Available: Support 5x churn in Azure Site Recovery Update #4 - Generally Available: Microsoft Entra ID-based access for Azure Blob Storage SFTP Update #1 - Generally Available: Network Security Perimeter support for Azure Event Hubs Why ITPros should care Network Security Perimeter for Event Hubs changes how ITPros enforce connectivity boundaries around mission-critical event pipelines. Instead of depending only on isolated firewall rules per namespace, you can apply perimeter-aware controls that are easier to govern consistently across multiple services. From an operations perspective, this is a service-level hardening improvement. It helps reduce accidental exposure and supports better audit conversations when security teams ask for clear evidence of allowed and denied paths. Operational value The operational value is stronger day-two control. You can standardise network access policy patterns for producer and consumer applications, reduce policy drift, and simplify incident investigations when unexpected traffic appears. For production rollout, validate all dependencies first: private endpoints, DNS resolution, trusted service exceptions, managed identities, and cross-subscription network paths. Real-world example with step-by-step guidance Inventory current producer and consumer traffic flows, including private endpoints, DNS zones, and any trusted service allowances. Deploy a pilot Event Hubs namespace with perimeter controls in non-production and mirror realistic ingestion and consumption traffic. Apply least-privilege inbound and outbound perimeter rules, then execute end-to-end send/receive tests with representative message volume. Review diagnostic logs for denies, refine exceptions only where business-justified, and capture evidence for change management. Promote to production in stages with a rollback plan that restores previous network policy if message flow health degrades. Technical details including code examples Use the following sequence when validating that perimeter onboarding did not break data plane operations. The first command confirms your active Azure context, the second verifies endpoint reachability, and the third validates Event Hub metadata retrieval. Run this safely in a test window before production enforcement. If connectivity and control-plane checks pass in test, repeat with production namespace read-only checks before enabling stricter policies. az account show --output table Test-NetConnection <namespace>.servicebus.windows.net -Port 5671 az eventhubs eventhub show --resource-group <rg> --namespace-name <namespace> --name <eventhub> --output table Expected outcome: TCP probe to port 5671 succeeds, and Event Hub metadata query returns without auth or network timeout errors. If probe fails, check DNS, NSGs, route tables, private endpoint linkage, and perimeter rule assignment scope. Comprehensive Resources Azure update: Network Security Perimeter support for Azure Event Hubs Network Security Perimeter concepts Azure Event Hubs documentation Event Hubs networking and security Update #2 - Generally Available: Confidential Computing support for Azure Event Hubs Dedicated Why ITPros should care Confidential Computing support for Event Hubs Dedicated matters when ITPros operate regulated or high-sensitivity event streams. It extends protection expectations beyond encryption at rest and in transit, into stronger assurances during processing. Compared with older architectures, this reduces the need for some compensating controls and helps security and operations teams align on platform-native protections for streaming workloads. Operational value Operationally, this strengthens trust boundaries for event ingestion platforms that feed analytics, SIEM, and business-critical automation. It also improves evidence posture for compliance reviews where data handling controls must be demonstrated end to end. Before rollout, validate throughput impact, partition behaviour, client compatibility, and observability baselines so confidentiality controls do not create unexpected SLO regressions. Real-world example with step-by-step guidance Classify Event Hubs namespaces by sensitivity and select the first dedicated environment where enhanced confidentiality requirements apply. Enable and validate in non-production with representative producer and consumer load, including peak and burst patterns. Measure latency, throughput, and throttling trends before and after enablement to confirm workload behaviour remains acceptable. Capture attestation and configuration evidence required by internal security governance or external auditors. Roll out in waves by workload criticality, with rollback criteria tied to message latency, error rates, and throttling thresholds. Technical details including code examples This validation example confirms namespace details and metrics health so you can compare baseline vs post-change behaviour. The metrics query focuses on ingestion, egress, and throttling signals that commonly surface operational risk first. Run with a least-privileged operations identity that can read namespace configuration and metrics. Avoid making unrelated changes while collecting baseline evidence. az eventhubs namespace show --resource-group <rg> --name <namespace> --output jsonc az monitor metrics list --resource /subscriptions/<sub>/resourceGroups/<rg>/providers/Microsoft.EventHub/namespaces/<namespace> --metric IncomingMessages OutgoingMessages ThrottledRequests --interval PT5M az account show --query user.name -o tsv Expected outcome: namespace query succeeds, metrics return consistently, and no abnormal throttling spike appears after control changes. If results diverge, review dedicated capacity planning, partition strategy, RBAC scope, and workload profile fidelity. Comprehensive Resources Azure update: Confidential Computing support for Azure Event Hubs Dedicated Event Hubs Dedicated overview Azure Confidential Computing overview Monitor Azure Event Hubs Update #3 - Generally Available: Support 5x churn in Azure Site Recovery Why ITPros should care Higher churn support in Azure Site Recovery is directly relevant for ITPros protecting write-intensive systems. It expands what can be replicated reliably, reducing DR exceptions for fast-changing workloads. Compared with the previous operational envelope, this gives more room for modern transactional applications while still requiring disciplined capacity and replication health management. Operational value Operational value is improved DR coverage and better alignment between production write behaviour and recovery plans. Teams can protect more workloads without bespoke workaround architecture. For production rollout, validate process server sizing, bandwidth headroom, cache storage performance, and sustained replication lag during peak change windows. Real-world example with step-by-step guidance Baseline current churn and replication lag for candidate workloads to identify which systems benefit most from the increased support. Enable replication in a pilot for one high-churn workload and observe initial seeding and steady-state health. Run test failover and reprotect to verify recovery objectives and operational runbook completeness. Tune bandwidth and cache settings if lag increases during peak write intervals or backup overlap windows. Onboard additional workloads incrementally and use replication health gates before each expansion wave. Technical details including code examples These commands are relevant for validating actual recovery readiness instead of configuration-only status. They expose protected item health and support controlled failover rehearsal. Use a non-production network for test failover and document outputs so operations and business continuity stakeholders share the same readiness evidence. az site-recovery fabric list --resource-group <rg> --vault-name <vault> -o table az site-recovery protected-item list --resource-group <rg> --vault-name <vault> --fabric-name <fabric> --protection-container <container> -o table az site-recovery recovery-plan test-failover --resource-group <rg> --vault-name <vault> --name <recoveryPlan> --network-id <testNetworkId> Expected outcome: protected items remain healthy, lag remains within target, and test failover completes without consistency errors. If failures occur, inspect connectivity, process server capacity, cache throughput, and policy mappings. Comprehensive Resources Azure update: Support 5x churn in Azure Site Recovery Azure Site Recovery documentation Monitor and troubleshoot Site Recovery Site Recovery capacity planning Update #4 - Generally Available: Microsoft Entra ID-based access for Azure Blob Storage SFTP Why ITPros should care This launch modernises SFTP access for Azure Blob Storage by bringing identity control closer to Microsoft Entra. ITPros gain stronger governance options than local-account-only models for many enterprise scenarios. Operationally, the key change is identity lifecycle alignment: provisioning, review, and revocation can be managed with central identity processes instead of fragmented local credentials. Operational value The value is reduced credential sprawl, better auditability, and clearer access accountability across teams and external partners exchanging files over SFTP. Before production, validate client compatibility, RBAC scope, network restrictions, access review cadence, and emergency break-glass procedures. Real-world example with step-by-step guidance Confirm SFTP is enabled on the storage account and validate networking model (public endpoint restrictions or private access path) matches policy. Assign Entra-based permissions with least privilege and validate scope at storage account and container boundaries. Test SFTP authentication and file operations using approved clients while collecting diagnostic logs for audit evidence. Validate joiner-mover-leaver scenarios by changing membership and role assignments, then confirming access updates propagate correctly. Roll out in stages by partner or workload segment with clear support ownership and incident response runbooks. Technical details including code examples This sequence verifies account capability and role assignment posture before user acceptance testing. It is useful for catching scope mistakes that often cause authentication-success/data-access-failure patterns. Run safely by using a dedicated test identity and non-production storage account first; then repeat read-only validation in production before broad enablement. az storage account show --name <storageAccount> --resource-group <rg> --query "{name:name,isSftpEnabled:isSftpEnabled,allowBlobPublicAccess:allowBlobPublicAccess}" -o jsonc az role assignment list --assignee <principalObjectId> --scope /subscriptions/<sub>/resourceGroups/<rg>/providers/Microsoft.Storage/storageAccounts/<storageAccount> -o table az account show --query user.name -o tsv Expected outcome: SFTP capability is enabled, expected role assignments are present, and test identity can perform allowed operations only. If sign-in works but file actions fail, inspect RBAC propagation delay, ACL/permission scope, and storage network restrictions. Comprehensive Resources Azure update: Microsoft Entra ID-based access for Azure Blob Storage SFTP SFTP support for Azure Blob Storage Authorize blob data with Microsoft Entra ID Azure Storage security baseline If you are planning adoption, start with one workload per update area, collect operational evidence, and standardise the validated pattern in your runbooks and IaC modules. That approach keeps change safe while accelerating delivery. Cheers! PierreStop Hand-Building VMs at 2 AM: Automated Image Pipelines with Azure Image Builder and Compute Gallery
Hello Folks! If you have ever stood up a marketplace Ubuntu VM, SSH’d in, layered on your monitoring agent, security tooling, a couple of CA certs, and a hardening script, then captured the result and called it your “golden image,” I have bad news. That image was already drifting from the next one your coworker built before you finished naming the snapshot. At the Microsoft Azure Infra Summit 2026, Sandeep Raichura (PM for Azure Compute Gallery) and Kofi Forsen (PM for Azure VM Image Builder) rebuilt the whole workflow the right way. Source, customize, validate, distribute, deploy. No clicks. No tribal knowledge. No 2 AM heroics. 📺 Watch the session: Why IT Pros Should Care You carry the pager when a bad image rolls into ten regions. You explain why three teams have three different Ubuntu 22.04 baselines with three different agents. You find out at 2 AM that someone deleted “the old image” and the old image was the one production VMSS was still pulling. This session is in your lane. It covers: Why hand-rolled images stop working the moment a second team needs one. How Azure VM Image Builder (AIB) turns image creation into declarative pipeline code. How Azure Compute Gallery handles versioning, replication, sharing, and accidental-deletion protection. How automatic image creation triggers chain a marketplace update through your golden image, into every downstream image, with zero manual steps. How VM Scale Sets close the loop with rolling upgrades and automatic OS upgrade. In short, this is the practitioner version of “do VM image management properly,” from the PMs who own both services. What is Azure Image Builder and Azure Compute Gallery The two services do different jobs and you really do need both. Azure VM Image Builder is the build engine. You hand it a JSON template that declares: A source (marketplace image, managed image, VHD, or existing gallery version). Customizers (shell, PowerShell, Windows updates, file copies, restart steps). One or more distribute targets (usually a Compute Gallery image definition). AIB spins up a temporary build VM, runs your customizers in order, validates, generalizes, captures, and publishes. Every build runs the exact same way. No SSH, no RDP, no “I forgot to install the monitoring agent this time.” Azure Compute Gallery is the management layer for the resulting artifacts. Formerly Shared Image Gallery, it has three levels: Gallery. The top-level container. Sharing policy lives here: RBAC, Direct Shared Gallery, or Community Gallery. Image definition. The metadata. OS type, generation, security type, publisher / offer / SKU. The SKU of an image family. Image version. The actual replicated artifact. Controls regions, replica counts, storage type (ZRS by default), end-of-life date, and the safety flags. AIB writes the artifact. Compute Gallery stores, versions, replicates, and shares it. Building an automated image pipeline The session walked through the five steps a real pipeline needs, with no manual intervention in the critical path: Source. A marketplace image or any other base. Customize. Scripts that install agents, harden, configure, and validate. Stored in a storage account so AIB can pull them with the right managed identity. Validate. Built-in validation hooks plus your own smoke tests baked into the customizer. Fail fast. Do not silently continue. Distribute. Push the captured image to a Compute Gallery image definition. Pick your regions and replica counts here. Version. Compute Gallery handles semantic versioning, replication, and safety flags. The trick that makes this a real pipeline is the two-template pattern Kofi demoed: A source template builds the org-wide golden image from the marketplace base. Its source reference is set to latest for the marketplace SKU (for example, Canonical Ubuntu 22.04 latest). A distro template layers user-group-specific tooling on top of the golden image. Its source reference is the golden image gallery version, also set to latest. Both templates get an automatic image creation trigger attached. Triggers only fire when the template references latest. From that point on: Canonical publishes a new Ubuntu 22.04. The source template’s trigger fires, AIB rebuilds your golden image, and a new version lands in the source gallery. That new golden image version fires the distro template’s trigger. AIB rebuilds every downstream distro image automatically. VM Scale Sets configured for automatic OS upgrade pick up the new image version and roll it out in batches, pausing if the Application Health probe goes red. You set it up once. After that you only come back when you want to change something on purpose. Safety by design in Compute Gallery A bad image at the top of this chain takes out thousands of VMs at the bottom. Sandeep was clear: safety is not optional, it is built in. The four features worth turning on every time: ZRS storage by default. Image versions stored on zone-redundant storage so a zonal failure does not take the image down. Exclude from latest. Stage an image into a region without making it the default for new deployments. Flip the flag when you are ready to roll. You can set this globally on the version or per region. Block deletion before end-of-life. The image cannot be deleted until its end-of-life date. This is the flag that stops the 2 AM accidental delete. Soft delete. If everything else fails, soft delete gives you a recovery window to restore an image version that should not have been removed. Combine those four with a sane end-of-life date on every version and your blast radius drops dramatically. Real-world scenarios A few patterns that came up in the session and the Q&A: Multi-region fleets. Define your target regions in the AIB template. AIB hands the artifact to Compute Gallery and Compute Gallery does the replication. Your scale sets in every region pull a local replica, not a cross-region copy. Open-source publisher. Use a Community Gallery so anyone in Azure can deploy your image. You provide a contact URL and email at the gallery level so consumers know where to file issues. Partner sharing. Use Direct Shared Gallery to grant specific subscriptions or tenants access without making the image public. VM Scale Sets with rolling upgrade. Reference the image definition (not a specific version) when you create the scale set. The scale set tracks latest. Pair it with a rolling upgrade policy and the Application Health extension. AIB publishes, Compute Gallery replicates, the scale set rolls, and the rollout pauses itself if the Application Health probe goes red. Getting Started Pick the highest-pain item and start there. You do not have to do this all at once. Stand up a Compute Gallery in one region. Create one image definition with proper publisher / offer / SKU metadata. Turn on soft delete at the gallery. Wrap an existing build script in an AIB image template. Use a marketplace image as the source. Distribute to your new gallery. Add excludeFromLatest, endOfLifeDate, and the block-deletion flags to your image version. Default to ZRS storage. Register the Microsoft.VirtualMachineImages and the triggers feature. Attach an automatic image creation trigger to the template. Set the source reference to latest. Build a second template that takes your golden image as its source. Attach a trigger to that one too. Create a VM Scale Set that references the image definition and enable automatic OS upgrade with rolling upgrades and the Application Health extension. That is the loop. Source updates flow through automatically. Bad images do not delete each other. Fleets roll forward in batches. Resources Azure VM Image Builder overview. The service concepts, supported OS, regions, and capabilities. Azure Compute Gallery overview. Gallery, definition, version, replication, and sharing. Azure VM Image Builder best practices. Identity, networking, customizers, and operational guidance from the product team. Automatic Image Creation with Image Builder triggers. Step-by-step to wire up source-image triggers. Create an image definition and image version. Portal, CLI, PowerShell, and REST flows for publishing artifacts. Automatic OS image upgrades for VM Scale Sets. The closing leg of the pipeline. Share images using Community Gallery. Public, non-commercial sharing for open-source publishers. Azure Image Builder samples on GitHub. Reference templates, customization scripts, and end-to-end examples. Watch the rest of the Summit This session was one of many at the Microsoft Azure Infrastructure Summit 2026. If you want the keynotes, the IaC deep dives, the AKS sessions, and the rest of the infra track, the full playlist is here: Microsoft Azure Infra Summit 2026 playlist Cheers! Pierre RomanLock Down AKS End to End with Application Gateway for Containers and Managed Cilium L7
Hello Folks! If your AKS cluster looks like most production clusters I have walked through, one of two things is true. Either nobody writes any network policies and every pod can talk to every other pod, so one compromised container blows up the entire blast radius. Or somebody wrote a few coarse rules along the lines of “namespace A talks to namespace B over port 80”, which sounds secure right up until an attacker realizes that port 80 is exactly where they were planning to live anyway. Real attacks happen at Layer 7, dressed up like ordinary HTTP traffic, and L3 / L4 plumbing cannot tell the difference. That is the gap session MAIS09 from the Microsoft Azure Infrastructure Summit 2026 closes. Vyshnavi Namani and Darshil Shah from the Azure Networking product team walked through how two AKS-managed add-ons, Application Gateway for Containers (AGC) and Cilium L7 via Advanced Container Networking Services (ACNS), can lock down the entire path from the public internet to a single pod. No NGINX. No external WAF appliance. No third-party CNI to babysit. 📺 Watch the session: Why IT Pros Should Care Let me cut to the chase. If you operate AKS clusters today, this session matters because: You probably still have an ingress controller and an external WAF stitched together with annotations and prayers. AGC plus ACNS collapses that stack into first-party add-ons that AKS owns end to end. Both Application Gateway for Containers and Advanced Container Networking Services are generally available. This is not a preview demo, this is production. Security is finally readable. Every rule is a YAML object. Code review, audit, GitOps. No more “what does this NGINX config map even do anymore” archaeology. It actually works on a real attack pattern. The demo shows WAF killing a SQL-injection-style GET that Cilium would have happily forwarded, because the method (GET) was on the allow list. If you have ever had to explain to an auditor why a single compromised pod could pivot across your whole cluster, this is your exit ramp. The AKS Security Gap This Closes Most clusters are protected by a load balancer at the edge and basically nothing inside. The cluster door looks like a vault, but the hallways are wide open. Cilium calls this the lateral movement problem, and it is exactly how Kubernetes attacks unfold in the wild. Compromise a pod, then phone home, then pivot. What MAIS09 demonstrates is something different. AGC is the L7 front door (the metal detector at the lobby). ACNS Cilium L7 is the lock on every pod’s office door. Both speak HTTP. Both enforce identity. Both are managed by AKS itself. The legacy alternative, Application Gateway Ingress Controller (AGIC), bolted a full Application Gateway onto your cluster through a translator. Two services, two lifecycles, two finger-pointing teams when something broke. AGC is the successor, built from scratch for Kubernetes, speaking the Gateway API natively, enabled with a single AKS flag. AKS provisions the controller, wires the identity, delegates the subnet, and owns the upgrades. You own the policies. AGC + Managed Cilium, End to End Here is the mental model from the session. Picture four concentric layers of defense between the public internet and a pod. AGC front end. One Azure resource, one public DNS name, and (thanks to the Kubernetes Gateway API) multiple hostnames behind the same IP. The demo runs Contoso, Fabrikam, and Adventure Works on a single AGC public IP using three HTTPRoute objects. One infrastructure, three websites. Real cost savings, real ownership clarity (platform owns the Gateway, app teams own the HTTPRoutes). Azure WAF on AGC. This is the content inspector. It runs the OWASP Core Rule Set (DRS 2.1 in the demo) against every incoming request, looks for SQL injection, cross-site scripting, path traversal, and the rest of the OWASP Top 10, and returns a 403 before the packet ever touches your pod. Microsoft maintains the rule set, you bind it to AGC via a SecurityPolicy. ACNS Cilium L7 ingress on every pod. This is where identity-based policy lives. Rules key off pod labels, not IPs, because IPs change every time the cluster autoscaler does its job. The demo uses an allow-agc-l7-get-only CiliumNetworkPolicy that lets the AGC backend reach the tenant pods, but only with GET or GET /products. Anything else, POST, PUT, DELETE, gets a Cilium-synthesized 403 before NGINX ever sees the request. ACNS east-west and egress policy. Two more policies do the heavy lifting inside. client-may-call-contoso-get-only lets the client pod reach Contoso with GET, and only Contoso. A default-deny baseline blocks everything else (pod-to-pod and pod-to-internet) with a single carve-out for kube-dns on port 53. The magic is that the same Cilium engine handles north-south, east-west, and egress with one consistent identity model. eBPF in the Linux kernel does the enforcement on the same node as the pod, so the decision happens before the packet leaves the host. No sidecars, no iptables sprawl, no daemonset you need to upgrade by hand. Real-world Scenarios The demo walks through six tests and the results map directly onto things you are probably trying to solve right now: Multi-site hosting on one IP. Three hostnames, one AGC, three 200 OKs from three different backend pods. If you are paying for three load balancers today, you can stop. WAF blocks a malicious GET that ACNS would have let through. This is the punch line of why you need both layers. The method (GET) is on the Cilium allow list, but the payload is a SQLi pattern. WAF returns 403 at the edge. Defense in depth, working as advertised. Method enforcement at the pod door. GET returns 200, POST/PUT/DELETE return 403, GET /admin returns 403, GET /products returns 200. Cilium is doing actual HTTP inspection, not just dropping packets. East-west enforcement with readable verdicts. Client to Contoso GET is 200. Same client, same destination, POST is 403 (L7 deny, TCP completed). Client to Fabrikam is 000 (L4 drop, no TCP handshake). Reading the difference between 403 and 000 is now a debuggable signal, not a mystery. Default-deny egress kills phone-home. A pod tries to reach bing.com. DNS resolves (the carve-out works), TCP SYN goes nowhere, wget gives up with exit code 1. If that pod was compromised and trying to exfiltrate data, this is where the attack chain dies. Selective allow still works. Same pod, same tools, but a DNS lookup against kube-dns inside the cluster returns instantly. We did not unplug the network. We locked it down with a purpose. Honest tradeoffs to call out. The session does not pretend everything is free. AGC introduces a billed subnet association and a managed identity you do not manage in BYO mode. Cilium L7 needs the Cilium data plane (ACNS Container Network Security features are Cilium-only). The Envoy proxy that handles L7 inspection has a cost only when you actually enforce L7, which is a fair deal in my book. Getting Started If you want to try this on a cluster of your own, three flags do most of the work on az aks create: --network-dataplane cilium (turns on the eBPF data plane) --enable-acns (enables Advanced Container Networking Services, including Hubble observability and Cilium L7 policy) --enable-app-routing or the ALB add-on flag (provisions the AGC controller as an AKS-managed add-on) From there you write four YAML objects: a default-deny CiliumNetworkPolicy, an allow-DNS carve-out, an AGC ingress allow with method and path constraints, and your east-west allow rules. The session repo includes the full set so you can clone and follow along. One bonus worth knowing about. ACNS ships Hubble out of the box, with pre-built Azure Managed Grafana dashboards. Flow logs, service maps, policy hit counts. Even on pods that are not yet under L7 enforcement, you get observability for free. When something breaks at 2 a.m., you have an audit trail instead of a tcpdump. Resources Azure Application Gateway for Containers documentation Set up Layer 7 policies with Advanced Container Networking Services AKS security concepts Cluster security best practices for AKS Container Network Observability for AKS (Hubble, Prometheus, Grafana) Advanced Container Networking Services hands-on lab Use cases of Advanced Network Observability for AKS (Azure Networking Blog) Watch the Rest of the Summit If MAIS09 hit the spot, there are dozens more sessions in the same playlist covering AKS networking at scale, Azure Local, AVM, the new Deployment Agent, and a lot more. Grab a coffee and binge. Microsoft Azure Infrastructure Summit 2026 playlist Cheers! Pierre RomanDeployment Stacks: Treating Your Azure Deployments Like Real Resources
Hello Folks! If you have ever come back from vacation to find a “mystery” storage account no one remembers creating, or watched a junior admin nuke a production resource because they clicked the wrong tile in the portal, this session is for you. Sydney Smith from the Azure Deployments team walked through Deployment Stacks at the Microsoft Azure Infrastructure Summit 2026, and the message is simple: classic Azure deployments are just history records, and history records cannot protect anything. Deployment Stacks change that. A stack is a real Azure resource. It has a resource ID. It supports RBAC. It tracks what it owns. It can refuse to let people break it. That is a real upgrade to how we run Infrastructure as Code in Azure, and IT pros should be paying attention. 📺 Watch the session: Why IT Pros Should Care Let me cut through the noise. Here is why this matters to anyone running Azure infrastructure: Orphaned resources cost real money. Classic deployments do not track ownership. Remove a resource from your template, and the live resource sits in Azure forever, billing you and adding security risk. Deployment history is fragile. Anyone can groom it away. You lose the audit trail. There is no native change protection. Without stacks, the only thing standing between your production VNet and a midnight portal click is RBAC discipline. That is not enough. Drift is invisible. You cannot detect what you cannot define as the source of truth. No unified view. Resources scattered across resource groups or subscriptions are hard to manage as one unit. Stacks give you a single lifecycle aware unit, with built in guardrails, that you can govern with RBAC and policy like any other Azure resource. That is the kind of plumbing IT operations has been asking for. What are Deployment Stacks A deployment stack is a native Azure resource that manages a group of resources as one lifecycle aware unit. It is the modern evolution of Azure deployments. Where a classic deployment is a record of what happened, a stack is a living thing that knows what it owns. Key facts to anchor on: A stack is of type Microsoft.Resources/deploymentStacks. It is a first class Azure resource. A stack can span multiple resource groups and even multiple subscriptions. You feed it a Bicep or ARM JSON template, the same files you already have. No rewrite required. It produces a snapshot of the resources it manages, and uses that snapshot for lifecycle decisions. Two feature sets do the heavy lifting: Resource lifecycle management. Creates, updates, and deletes across scopes are managed as one operation. Resource change protection. Stacks can apply deny assignments on the resources they manage, blocking changes that did not come through the stack itself. That second point is the one that changes how you think about safety in production. How Deployment Stacks change your workflow Honestly, the syntax change is tiny. If you know az deployment group create, you already know az stack group create. Same template, same parameter files, same scope. You just swap deployment for stack and add a couple of new parameters. What changes in the day to day: Your template becomes the source of truth. Remove a resource from Bicep, and the next stack run can clean it up automatically. No more manual hunt and delete. You get a portal view. The stack resource shows you what it manages, what is detached, and what the deny settings look like. You can clean up everything in one step. Deleting a stack with the right unmanage setting tears down all its resources and the empty resource groups, without you mapping out dependency order. Pipelines get cleaner. The bicep-deploy GitHub Action and the equivalent Azure DevOps task speak stacks natively. Create, validate, what-if, delete. Same shared package on the backend, so features hit GitHub and ADO at roughly the same time. The honest tradeoff: stacks are powerful enough to delete things at scale. That is the whole point. So you need to think harder about which mode you pick. Deny settings and lifecycle control Two parameters carry most of the weight, and you need to understand both before you push a stack into production. action-on-unmanage controls what happens when a resource leaves the template: detachAll: resources stay in Azure, the stack stops tracking them. This is the default and the safest. No delete operation. Good when you are still learning stacks. deleteResources: resources are deleted, the resource group stays. deleteAll: full cleanup. Resources go, empty resource groups go. This is what you pair with strict deny settings if you want the template to be the only source of truth. deny-settings-mode controls who can change what: none: no protection. Anyone with RBAC can do anything. Use this only when you really mean it. denyDelete: blocks delete operations. Updates are still allowed. denyWriteAndDelete: the strict one. No changes outside the stack. This plus deleteAll makes your template the absolute source of truth. A handy escape hatch: excludedActions lets you punch specific resource types or operations through the deny wall. For example, you can deny delete across the board but allow deleting your managed identities or storage accounts. Use this sparingly. Every exception is a future ticket. A few honest tradeoffs to keep in mind: denyWriteAndDelete is strict on purpose. Portal admins will get permission errors. That is the feature, not a bug. Train your team. deleteAll is permanent. Pair it with what-if before every change so you do not nuke something by accident. Resources should not be managed by two stacks. The deny assignments from each stack will fight each other and block updates. If two apps share a SQL database, pull it in as an existing resource rather than dual managing it. Stacks are not just for Bicep. They consume ARM JSON templates today. Terraform users do not get the same first party stack experience, so if you live in Terraform world, stacks are not your tool. Getting Started You do not need a giant rollout plan. Pick something small and learn the parameters in a safe environment. A practical path: Pick a non production Bicep template you already trust. Anything that creates a small set of resources in one resource group works. Deploy it as a stack with safe defaults. Use actionOnUnmanage=detachAll and denySettingsMode=denyDelete. You get protection without risking accidental deletes. Try removing a resource from the template and redeploy. Watch how the resource shows up as detached in the portal. Get comfortable with what that looks like. Graduate to deletes. When you trust the workflow, switch to deleteResources or deleteAll in a dev environment. Always run what-if first. Adopt Stacks What-If. The newer Stacks What-If commandlet triangulates the stack snapshot, the live resource state, and the desired state from your template. The result is a far less noisy diff than classic what-if, and the RBAC model only needs read access on the what-if resource for follow up gets. Big win for production review workflows. Wire it into your pipeline. The bicep-deploy GitHub Action and the Azure DevOps task both support type: deploymentStack with create, validate, and delete operations. Start with a create pipeline, then add a what-if pipeline for PR validation, then a cleanup pipeline that uses delete to tear down ephemeral environments. If you hit an out of sync error between your stack and Azure, the bypass-stack-out-of-sync-error parameter lets the pipeline reconverge automatically instead of forcing a manual cleanup. Read the docs before you flip it on, but it saves real time when state drifts. Resources Create and deploy Azure deployment stacks in Bicep Bicep documentation on Microsoft Learn What-If for ARM and Bicep deployments Deploy Bicep files by using GitHub Actions Azure Bicep on GitHub Bicep Deploy GitHub Action and ADO task Bicep and ARM community news and monthly call signup Watch the rest of the Summit If this session helped, there are plenty more where it came from. The full Microsoft Azure Infrastructure Summit 2026 playlist covers everything from Azure Local and Azure Files to AKS networking, SRE agents, and the keynote with Jeremy Winter. Worth a binge if you run Azure infrastructure for a living. Microsoft Azure Infrastructure Summit 2026 playlist Cheers! Pierre Roman
