Microsoft Sovereignty 2026: From Data Residency to Digital Control
Over the past few years, data sovereignty has evolved from a compliance checkbox to a board-level priority. What began as a discussion around where data is stored has now expanded to who controls it, who operates it and under which jurisdiction it is governed. As we move into 2026, Microsoft Sovereignty is no longer just a roadmap, it is actively shaping how enterprises design cloud and AI architectures, especially across regulated industries. Why Sovereignty Matters More Than Ever Organizations today are navigating a complex landscape: Increasing regulatory mandates (GDPR, NIS2, DORA) Rising geopolitical concerns around cross-border data access Accelerated adoption of AI, copilots, and agentic systems But what’s changing in 2026 is the scale of AI adoption: 1.3B AI agents expected by 2028 82% of organizations plan to integrate AI agents within 1–3 years 90% of developers will use AI-assisted coding tools This fundamentally shifts the sovereignty discussion: It’s no longer about protecting data, it’s about governing AI-driven decisions and automation. Sovereignty in the Age of AI Agents A critical insight emerging from the field: Not all AI workloads can run in public cloud environments. Some AI scenarios require sovereignty by design, especially when: Data must remain within national jurisdiction Operational access must be restricted Systems must continue functioning during disconnection or crisis Examples include: Government AI copilots for citizen services Defense systems requiring air-gapped AI Financial services with strict regulatory oversight Healthcare workloads with sensitive patient data AI strategies must now survive regulation, disruption and disconnection not just scale. Microsoft Sovereignty: A Multi-Layered Approach Microsoft’s approach to sovereignty is not a single feature it’s a comprehensive framework spanning infrastructure, operations, security and AI. At its core, Microsoft Sovereign Cloud introduces three key deployment models: 1. Sovereign Public Cloud Regional data boundaries and in-country processing Built-in sovereign controls at hyperscale AI model choice with localized processing 2. Sovereign Private Cloud (AI-Driven Evolution) This is where sovereignty is evolving the fastest in 2026. Runs on Azure Local + Microsoft 365 Local + Foundry Local Enables continuous operations in hybrid or disconnected environments Supports AI workloads with local inferencing and GPU acceleration This is no longer traditional on-prem it is cloud-grade AI deployed locally. 3. National Partner Clouds Operated by local entities Meets country-specific certifications Bridges global cloud and national regulations Sovereign AI: From Data Control to Full Lifecycle Control The biggest shift in 2026: Sovereignty is no longer just about data it’s about the entire AI lifecycle. Sovereign AI ensures: Data stays local and under customer authority AI systems operate even without connectivity Customers control model selection (proprietary, OSS or custom) This introduces a new dimension: Model Sovereignty + Operational Sovereignty + Infrastructure Sovereignty The Rise of Foundry Local: AI From Cloud to Edge One of the most important innovations enabling this shift is Microsoft Foundry Local. Foundry Local extends AI capabilities across: Cloud Edge devices On-premises environments Fully disconnected deployments This allows organizations to: Run models locally using containers Use Arc-enabled Kubernetes for deployment Maintain consistent governance across environments AI Models Under Sovereign Control Microsoft enables multiple AI model strategies: Models-as-a-Platform (MaaP) → Customer-managed Models-as-a-Service (MaaS) → Microsoft-managed BYO Models → Full flexibility (Open-source or proprietary) This means enterprises can shift from: ❌ Vendor-dependent AI ✅ Sovereign, customer-controlled AI ecosystems Sovereign AI Deployment Patterns Two dominant patterns are emerging: 1. Hybrid Sovereign AI Develop in cloud Deploy to edge or sovereign environments Maintain flexibility 2. Fully Disconnected AI Air-gapped environments No dependency on cloud connectivity Full local processing and inference This is critical for defense, public sector and critical infrastructure. The Reality Check: What Enterprises Must Still Own While Microsoft provides the platform, sovereignty is not “set and forget.” Organizations must still: Design region-first and sovereignty-aware architectures Implement governance across hybrid and disconnected environments Manage model lifecycle and inferencing policies locally Ensure compliance with evolving regulatory frameworks Sovereignty is now an architecture decision not just a cloud feature. My Perspective (Field Insight) From working with regulated customers (BFSI, telecom, public sector), I see three clear patterns: 1. Sovereignty is now directly tied to AI adoption → Customers will not scale GenAI without sovereign guarantees 2. Hybrid + Sovereign AI is becoming the default architecture → Cloud-only strategies are no longer sufficient 3. Control of models and inferencing is the new trust boundary → Trust is shifting from infrastructure to AI execution layers Final Thoughts: Sovereignty as an AI Enabler The narrative around sovereignty is shifting: ❌ Earlier: “Sovereignty restricts innovation” ✅ Now: “Sovereignty enables trusted AI at scale” Microsoft’s Sovereign Cloud strategy reflects this evolution bringing together: Cloud-scale capabilities Local control and resilience AI lifecycle governance The opportunity ahead is clear: Design sovereign-by-default AI architectures that are secure, compliant and built for resilience whether connected, hybrid or fully disconnected.Modernizing Terraform Pipelines on Azure: OIDC Federation for GitHub Actions and Azure DevOps
The secret nobody wants to rotate Most Terraform-on-Azure pipelines we see still authenticate the same way they did three years ago. A long-lived ARM_CLIENT_SECRET sitting in GitHub Actions or Azure DevOps, set once, copied around, and rotated only when something breaks. It's the most ignored credential in the cloud, and statistically the most likely one to leak. A developer screenshots a variable group. A pipeline log echoes a value. A fork inherits a secret. Or the secret simply expires on a Friday evening and takes production deployments with it. Workload Identity Federation (WIF) makes this whole class of problem go away. The pipeline mints a short-lived token at runtime, exchanges it for an Azure access token via Microsoft Entra, and never touches a secret. GitHub Actions has supported it since 2021. Azure DevOps service connections went GA with WIF in February 2024. The azurerm Terraform provider has supported it since v3.7. This post walks through the pattern end-to-end, for both GitHub Actions and Azure DevOps, the way I've rolled it out across multiple customer estates. How the exchange actually works Before any YAML, it helps to picture what's happening: The CI system (GitHub or ADO) signs a short-lived JWT describing exactly what's running- which repo, which branch, which environment, which service connection. The pipeline sends that JWT to Microsoft Entra ID. Entra checks it against a federated identity credential you've configured on a managed identity or app registration. The iss, sub, and aud claims must match case-sensitively. If it matches, Entra returns an Azure access token valid for the duration of the job. Terraform uses it. The job ends. The token expires. Nothing persists. The token is bound to a specific subject like repo:contoso/platform:environment:prod or sc://contoso/platform/azure-prod. It can't be reused from another repo, branch, or pipeline. Recommended Architecture A few choices that usually hold up in production: Decision Choice Identity type User-assigned managed identity (UAMI), not app registration Identity granularity One UAMI per environment (not per pipeline) Trust scope Pinned to the environment claim, not the branch RBAC scope Resource group, not subscription Remote state OIDC + use_azuread_auth = true, shared key access disabled Why UAMIs? They live in your subscription, don't need Application Administrator rights to manage, and follow the lifecycle of the resource group they belong to. Why one per environment? Pipeline-per-identity explodes into hundreds of identities. Environment-per-identity maps cleanly to deployment scopes. Part 1 - GitHub Actions Step 1: Create the identity and federate it Two commands per environment. That's it. az identity create -g rg-platform-identity -n id-tf-prod -l eastus az identity federated-credential create \ --name github-prod \ --identity-name id-tf-prod \ --resource-group rg-platform-identity \ --issuer https://token.actions.githubusercontent.com \ --subject repo:contoso/platform:environment:prod \ --audiences api://AzureADTokenExchange Repeat for nonprod. No secret is created anywhere. Step 2: Wire it up in GitHub In repo Settings → Environments, create nonprod and prod. On prod, add required reviewers and a branch rule restricting deployments to main. Then add three environment variables (not secrets - these aren't sensitive): AZURE_CLIENT_ID, AZURE_TENANT_ID, AZURE_SUBSCRIPTION_ID. The workflow itself stays small: permissions: id-token: write contents: read jobs: apply: runs-on: ubuntu-latest environment: prod env: ARM_USE_OIDC: "true" ARM_CLIENT_ID: ${{ vars.AZURE_CLIENT_ID }} ARM_TENANT_ID: ${{ vars.AZURE_TENANT_ID }} ARM_SUBSCRIPTION_ID: ${{ vars.AZURE_SUBSCRIPTION_ID }} steps: - uses: actions/checkout@v4 - uses: hashicorp/setup-terraform@v3 - run: terraform init && terraform apply -auto-approve Three things make this secure: id-token: write is the only elevated permission, and it doesn't grant write access to anything in GitHub, it just lets the runner mint a JWT. The environment: line picks the right AZURE_CLIENT_ID and drives the sub claim. The federation refuses anything else. No azure/login step is needed for Terraform. The azurerm provider reads GitHub's OIDC environment variables automatically. Part 2 - Azure DevOps The model is identical. The mechanics are different. ADO offers two creation paths for a WIF service connection: automatic (it creates an app registration for you) and manual (you bring your own UAMI). For platform teams, manual + UAMI is almost always the better choice to ensure identity lives where governance lives. The flow is a small dance between the two portals: In Azure DevOps, create a new ARM service connection → choose Workload Identity Federation (manual) → fill in your UAMI's client ID, tenant ID, and subscription. Save as draft. ADO shows you an issuer URL and a subject identifier. In Azure, on the UAMI, add a federated credential using the values ADO showed you. The subject looks like sc://contoso/platform/azure-prod. Back in ADO, click Verify and save. In the pipeline, the service connection only "activates" if a task in the job loads it. The simplest way is the AzureCLI@2 task: - task: AzureCLI@2 inputs: azureSubscription: azure-prod # the WIF service connection scriptType: bash scriptLocation: inlineScript inlineScript: | terraform init && terraform apply -auto-approve env: ARM_USE_OIDC: "true" ARM_CLIENT_ID: $(AZURE_CLIENT_ID) ARM_TENANT_ID: $(AZURE_TENANT_ID) ARM_SUBSCRIPTION_ID: $(AZURE_SUBSCRIPTION_ID) ARM_ADO_PIPELINE_SERVICE_CONNECTION_ID: $(SERVICE_CONNECTION_ID) SYSTEM_ACCESSTOKEN: $(System.AccessToken) SYSTEM_OIDCREQUESTURI: $(System.OidcRequestUri) For teams converting dozens of legacy connections, the Azure DevOps team published a PowerShell helper that walks every ARM service connection in a project and converts them in place. There's a 7-day rollback window on each connection, which makes the migration genuinely low-risk. Don't forget the state file The Terraform state is your real blast radius. With OIDC, it's almost free to lock it down too. The same UAMI can read and write blob data without the storage account key: backend "azurerm" { resource_group_name = "rg-tfstate" storage_account_name = "sttfstateprodeastus" container_name = "platform-prod" key = "platform.tfstate" use_oidc = true use_azuread_auth = true } Grant the UAMI Storage Blob Data Contributor on the container (not the account), disable shared key access on the storage account, and you've removed the last secret in the pipeline. RBAC and break-glass Federation removes a credential, not a privilege. A few habits worth keeping: Scope role assignments to resource groups, not subscriptions. The whole point of federation is that scoping is now trivially easy. Use Role Based Access Control Administrator instead of User Access Administrator if your Terraform creates role assignments. It's a more recent, narrower role. Have a documented break-glass. If GitHub or ADO has a token-service incident, you still need a path to ship a hotfix. A single hardware-key-protected emergency app registration in a separate identity boundary works well, audited monthly. Monitor sign-ins. Every federated exchange shows up in Entra sign-in logs as a service principal sign-in. Pipe these to Sentinel and alert on anomalies like sign-ins outside expected hours, or from IPs outside GitHub's published ranges. The errors you will hit (and what they really mean) Symptom What it actually is AADSTS70021: No matching federated identity record found Case-sensitive mismatch in iss, sub, or aud. Almost always a trailing slash or a capitalised character AADSTS700016: Application not found in directory Wrong client ID or tenant. Not a federation problem 403 on a resource even though token exchange worked Federation is fine. Your RBAC isn't. Check the exact scope Unable to determine OIDC token (ADO) No task in the job loaded the service connection. Add an AzureCLI@2 step Works on main, fails on tags You pinned sub to a branch ref. Add a second federated credential for tags, or move to environment-based scoping Migrating without a maintenance window You almost never get to do this on a greenfield repo. The order that has worked for me on legacy estates: Create the new UAMI alongside the old service principal, with the same role assignments. Federate one canary pipeline. Verify it deploys equivalently. Cut over pipelines in waves, lowest-risk environment first. Once a full release cycle passes cleanly, disable the old SP's secret. Wait another cycle. Then delete the SP entirely. Add a CI gate that fails any new pipeline introducing ARM_CLIENT_SECRET. The old and new auth methods coexist on the same subscription throughout. There's no hard cutover and no maintenance window, just a steady drift toward zero secrets. Wrapping up If you do nothing else after reading this, do one thing: search your CI variable groups for ARM_CLIENT_SECRET. Every result is an outage or a breach waiting to happen. Federation is one of those rare changes that's both more secure and less work to operate. Once you've set it up, you stop thinking about credential rotation, secret expiry, and quarterly access reviews for service principals. The pipeline simply runs, and the audit trail is in Entra where it belongs. That's a good trade.
CHERIoT-Ibex: Closing the door on memory safety vulnerabilities with hardware-enforced protection
Memory safety vulnerabilities—largely arising from widely used programming languages such as C and C++—remain a leading cause of exploitable software defects across systems, from embedded devices to cloud-scale infrastructure. In simple terms, memory safety ensures that software accesses only the data it is intended to use; when this protection fails, attackers can exploit these defects to gain control of devices or disrupt critical services. Industry data shows that about 70 percent of the vulnerabilities Microsoft assigns as Common Vulnerabilities and Exposures (CVE) each year are memory safety issues, highlighting how frequently these software defects translate into real-world security risk (CISA – The Urgent Need for Memory Safety in Software Products). Hardware-enforced protections such as CHERIoT-Ibex can help eliminate these vulnerabilities at their source, reducing the likelihood that low-level software flaws can be exploited to compromise devices or disrupt workloads, supporting more trustworthy infrastructure by design. An open and certified foundation for memory-safe embedded systems CHERIoT-Ibex is the first open-source production-quality implementation of the CHERIoT instruction set architecture and among the first cores certified by the CHERI Alliance (CHERI Alliance – CHERIoT). CHERIoT is an extension of the CHERI (Capability Hardware Enhanced RISC Instructions) instruction set, with a focus on embedded and Internet of Things (IoT) applications. Ibex is an open‑source 32‑bit RISC‑V core developed by LowRISC. CHERIoT‑Ibex builds on Ibex by including CHERIoT capability extensions to provide hardware‑enforced memory safety and fine‑grained compartmentalization. It is the result of a close partnership between Microsoft Research and Azure Hardware Systems & Infrastructure, combining advanced research innovation with industry-leading silicon IP development expertise. In 2023, Microsoft open-sourced the CHERIoT Platform to bring hardware-enforced memory safety to embedded systems, including an instruction set architecture, toolchain, real-time operating system, and the RTL implementation of the CHERIoT-Ibex core. The CHERI Alliance certification recognizes its ability to provide spatial and temporal memory safety, fine-grained compartmentalization, and compatibility with the broader CHERI ecosystem. Critically, CHERIoT-Ibex achieves these security guarantees with power and area efficiency comparable to low-cost microcontrollers, demonstrating that security doesn’t have to come at a premium. Why memory safety remains a foundational security challenge Traditional embedded and microcontroller-class designs rely on software hardening and coarse-grained hardware protections that struggle to prevent attacks such as buffer overflows and use-after-free vulnerabilities, often adding complexity while still leaving gaps in protection. Consider a controller that runs privileged firmware responsible for device initialization, telemetry, and system health monitoring, while also hosting networking functionality exposed to external inputs. A memory-safe vulnerability in the networking stack could allow attackers to execute unauthorized code within the firmware environment, potentially affecting other critical services on the device. In tightly integrated systems, these failures can propagate beyond a single component, increasing overall risk. Constraining failures with hardware-enforced isolation CHERIoT-Ibex enables hardware-enforced isolation between these components, helping ensure that even if the networking stack is compromised, its ability to impact system initialization or telemetry functions remains constrained. By limiting the blast radius of software failures, CHERIoT-Ibex supports a system-level approach to security rather than relying on individual components to defend themselves in isolation. Advancing memory-safe infrastructure by design CHERIoT-Ibex’s certification by the CHERI Alliance marks an important milestone for open-source memory-safe solutions. It validates that strong security guarantees can coexist with efficiency and transparency, reflecting Microsoft’s broader silicon-to-systems strategy of embedding security into the foundational hardware infrastructure. Explore and engage with the open-source CHERIoT ecosystem by visiting the CHERIoT Platform and the CHERIoT-Ibex GitHub repository (microsoft/cheriot-ibex). The repositories enable developers and researchers to experiment with, contribute to, and build on memory-safe hardware and software foundations.
Deploying Azure Redis Enterprise with Geo-Replication Using Terraform
This post walks through a production‑proven pattern for running stateful services across Azure regions using Terraform. We’ll cover a primary–replica Redis architecture, regional isolation with Key Vault and networking, and a clean Terraform parameterization strategy that scales from development to production without duplication. Why Multi‑Region State Is Hard Running applications globally is easy when everything is stateless—if something fails, you redeploy. But stateful services tell a different story. Caches, message brokers, and data stores can’t be treated as disposable. They hold business‑critical data, and downtime or inconsistency quickly becomes customer‑visible. In real‑world systems, common requirements include: Low‑latency reads from multiple regions Automatic recovery when a region becomes unavailable Predictable data consistency Repeatable infrastructure from dev through production Manually configuring this per region doesn’t scale. Drift sets in. Failover is unclear. Backups get forgotten. That’s where Terraform + Azure Managed Redis geo‑replication shines. Github Link : https://github.com/vsakash5/Managed-redis.git High‑Level Architecture We use a primary–replica Redis Enterprise model: Primary Redis Single write endpoint Highly available inside its region Source of truth Replica Redis Read‑only Asynchronously synced from primary Can be promoted during disaster recovery Each region is fully isolated: Separate subnets Separate Key Vaults Private Endpoints only (no public exposure) This prevents shared failure domains and allows each region to operate independently if needed. The Terraform Design Principle Instead of maintaining separate Terraform stacks per region, the key idea is: One reusable module, one tfvars file per environment, multiple regions inside it. The module is written once. Regional differences are supplied via parameter suffixes like: _replica _secondary _tertiary This keeps logic centralized and environments consistent. Core Parameter Layers 1. Environment Identity (Shared) Terraform environment = "dev" # dev | staging | prod context_prefix = "app" Show more lines These values are reused everywhere—names, tags, and identifiers. 2. Primary Region Terraform location = "eastus2" resource_group_name = "rg-app-dev-primary" Show more lines 3. Replica Region Terraform location_replica = "uksouth" resource_group_name_replica = "rg-app-dev-replica" The symmetry is intentional. Terraform can now apply the same module twice without branching logic. Regional Isolation: Networking and Secrets Why isolation matters Geo‑replication copies data, not dependencies. If both Redis instances depend on: the same subnet the same Key Vault then a failure in one region can cascade into the other. Networking (One Subnet per Region) Benefits: Independent NSGs Independent routing Independent capacity planning Key Vault (One per Region) Why this matters: Redis credentials are not replicated Each region stores its own secrets A Key Vault outage doesn’t take both regions down Redis Configuration Primary Redis (Writes Enabled) The geo‑replication group name must match. That’s the logical binding Azure uses to link instances. Private Endpoint‑Only Access No Redis instance is exposed publicly. Each region uses: A private endpoint A workload subnet Internal DNS resolution This means: No public IPs No inbound attack surface Traffic stays on the Azure backbone Linking Primary and Replica Terraform explicitly defines the relationship: Terraform managed_redis_geo_replication_config = { primary_to_replica = { primary_redis_key = "primary" replica_keys = ["replica"] } } Terraform ensures: Primary is created first Replica is deployed second Geo‑replication is established last Environment Scaling: Dev → Staging → Prod The infrastructure pattern never changes. Only values do. Environment Group Name Dev dev-grp Staging stg-grp Prod prod-grp This is how you avoid “snowflake” environments. Disaster Recovery Strategy If the primary region fails: Applications fail over to the replica read endpoint Terraform configuration is updated to: Remove geo‑replication Promote replica config to primary Traffic is fully restored Once the original region recovers, roles can be re‑established cleanly. No click‑ops. No guesswork. Key Lessons Learned 1. Naming is Infrastructure Predictable names enable automation, discovery, and auditing. 2. Key Vault Isolation Beats Availability A shared Key Vault is a shared outage. 3. Parameterization Beats Copy‑Paste Fix once → benefit everywhere. 4. Geo‑Replication Is a Contract Matching replication group names is non‑negotiable. 5. The tfvars File Is the Source of Truth If it’s not in Terraform, it’s not real. Final Thoughts Running stateful services in multiple regions doesn’t require magic— it requires discipline: Isolate aggressively Parameterize consistently Automate everything Test failure often With this approach, adding a new region becomes configuration—not redesign. That’s how infrastructure scales.Security Copilot Agents in Defender XDR: where things actually stand
With RSAC 2026 behind us and the E5 inclusion now rolling out between April 20 and June 30, anyone planning SOC workflows or sitting on a capacity budget needs to get a clear picture of what is GA, what is preview, and what was just announced. The marketing pages tend to blur those lines. This is my sober look at the current state, with the operational details that matter for adoption decisions. What is actually shipping right now The Phishing Triage Agent is GA. It only handles user-reported phish through Defender for Office 365 P2, but for most SOCs that is a meaningful chunk of the L1 queue. Verdicts come with a natural-language rationale rather than just a label, which is the part that determines whether analysts will trust it. The agent learns from analyst confirmations and overrides, so the feedback loop matters more than the initial setup. There is a setup detail that is easy to miss: the agent will not classify alerts that have already been suppressed by alert tuning. The built-in rule "Auto-Resolve - Email reported by user as malware or phish" needs to be off, and any custom tuning rules that touch this alert type need review. If you skip this, the agent runs on an empty queue and you wonder why nothing is happening. The Threat Intelligence Briefing Agent is also GA. It produces tenant-tailored intel briefings on a regular cadence. Useful, but lower operational impact than the triage agents. Copilot Chat in Defender went GA with the April 2026 update. Conversational Q&A inside the portal, grounded in your incident and entity data. This is the lowest-risk way to get value out of Security Copilot and probably where most teams should start. Public preview, worth watching The Dynamic Threat Detection Agent is the most technically interesting one. It runs continuously in the Defender backend, correlates across Defender and Sentinel telemetry, generates its own hypotheses, and emits a dynamic alert when the evidence converges. Detection source on the alert is Security Copilot. Each alert includes the structured fields (severity, MITRE techniques, remediation) plus a narrative explaining the reasoning. For EU tenants the residency point is worth confirming with whoever owns data protection in your org: the service runs region-local, so customer data and required telemetry stay inside the designated geographic boundary. During public preview it is enabled by default for eligible customers and is free. At GA, currently targeted for late 2026, it transitions to the SCU consumption model and can be disabled. The Threat Hunting Agent is also in public preview. Natural language to KQL with guided hunting. Lower stakes, but useful for teams without deep KQL expertise on hand. Announced at RSAC, still preview Two agents got the headlines in March: The Security Alert Triage Agent extends the agentic triage approach beyond phishing into identity and cloud alerts. The longer-term direction is consolidating phishing, identity, and cloud triage under a single agent. Rollout is from April 2026, in preview. The Security Analyst Agent is the multi-step investigation agent. Deeper context across Defender and Sentinel, prioritised findings, transparent reasoning trace. Preview since March 26. Both look promising on paper, but Microsoft's history of preview features that take a long time to mature is well-documented. I would not plan production workflows around either of them yet. What you actually get with the E5 inclusion This is the licensing change most people are dealing with right now. Security Copilot has been part of the E5 product terms since January 1, 2026. Tenant rollout is phased between April 20 and June 30, 2026, with a 7-day notification before activation. The numbers: 400 SCUs per month for every 1,000 paid user licenses Capped at 10,000 SCUs per month, which you hit at around 25,000 seats Linear scaling below that, so a 3,000-seat tenant gets 1,200 SCUs per month No rollover, the pool resets monthly What is included: chat, promptbooks, agentic scenarios across Defender, Entra, Intune, Purview, and the standalone portal. Agent Builder and the Graph APIs are in. If you also run Sentinel, the included SCUs apply to Security Copilot scenarios there. What is not included: Sentinel data lake compute and storage. Those still run through Azure on the regular meters. Beyond the included pool you pay 6 USD per SCU pay-as-you-go, with 30 days notice before that mode kicks in. Practical things worth knowing before activation A few details that are easy to miss in the docs: Under System > Settings > Copilot in Defender > Preferences, switch from Auto-generate to Generate on demand. Auto-generate will burn SCUs on incidents nobody is going to look at. Generate on demand gives you direct control. In the Security Copilot portal workspace settings, check the data storage location and the data sharing toggle. Data sharing is on by default, which means Microsoft uses interaction data for product improvement. If your compliance position does not allow that, change it before agents start running. Changing it requires the Capacity Contributor role. Agent runs are not equivalent to the same number of analyst chat prompts. A triage agent processing fifty alerts in one run consumes meaningfully more SCUs than fifty manual prompts on the same data. If you have a high-volume phishing pipeline, model that out before you flip the switch broadly. The usage dashboard in the Security Copilot portal breaks down consumption by day, user, and scenario. Output quality depends on telemetry quality. Flaky connectors, gaps in log sources, or a high baseline of misconfigured alerts will produce verdicts that match. Connector health monitoring (the SentinelHealth table in Advanced Hunting is a sensible starting point) is a precondition. The agents only improve if analysts feed the override loop. If your team treats the verdicts as background noise rather than confirming or correcting them, the feedback signal is lost and calibration stays where it shipped. That is a process problem, not a product problem, but it determines whether any of this is worth the SCUs. A reasonable adoption order A rough sequence that minimises capacity surprises: Copilot Chat in Defender first. Lowest risk, immediate value through natural language Q&A in the investigation context. Phishing Triage Agent on a controlled subset, with a review cadence in place. Check the built-in tuning rules first. Watch the SCU dashboard for the first month before adding anything else. Let the Dynamic Threat Detection Agent run while it is in public preview, since it is default-on and free anyway. Compare its alerts against existing Sentinel detections. Security Alert Triage Agent for identity and cloud once the phishing baseline is stable. Establish a monthly review covering agent decisions, false-positive rate, SCU cost, and MTTD/MTTR trends. Technically, agentic triage is moving past phishing into identity and cloud, and the Dynamic Threat Detection Agent represents a genuine attempt at the false-negative problem rather than just another rule engine. Lizenziell, the E5 inclusion removes the biggest barrier to adoption that previously existed. The risk is enabling everything at once. Agents that nobody reviews are agents that consume capacity without delivering value, and the SCU dashboard is the only thing that will tell you that is happening. One agent, one use case, a 30-day baseline, then the next one. The order matters more than the speed.Designing Outbound Connectivity for "Private Subnets" in Azure
Why Private Subnets Change Everything Historically, Azure virtual machines relied on default outbound internet access, where the platform automatically assigned a dynamic SNAT IP from a shared pool. This was convenient but problematic: ❌ No deterministic outbound IP addresses ❌ No traffic inspection or filtering ❌ No FQDN or URL governance ❌ Difficult to audit for compliance ❌ Susceptible to noisy neighbor SNAT exhaustion With private subnets, outbound access is disabled by default. This shifts the responsibility to the architect — deliberately. The result is an environment where: ✅ Every outbound flow is intentional ✅ Every outbound IP is known and documented ✅ Every egress path can be governed and logged ✅ Compliance evidence is straightforward to produce The question is no longer "does my VM have internet access?" but rather "how exactly does my VM reach the internet, and is that path appropriate for this workload?" The Three Outbound Patterns at a Glance Option Primary Role Inspection Scale Cost Best For NAT Gateway Managed outbound SNAT ❌ None ⭐⭐⭐ High 💲 Low Simple, scalable egress Azure Firewall Secure governed egress ✅ Full L3–L7 ⭐⭐⭐ High 💲💲💲 Higher Security boundaries Load Balancer Legacy SNAT ❌ None ⭐⭐ Limited 💲 Low Legacy / transitional Scenario 1: NAT Gateway What is NAT Gateway? Azure NAT Gateway is a fully managed, zone‑resilient, outbound‑only SNAT service. It attaches at the subnet level and automatically handles all outbound flows from that subnet using one or more static public IP addresses or prefixes. It is purpose‑built for one thing: providing predictable, scalable outbound internet access — without routing complexity or inline devices. Key flow are depicted below: VM → NAT Gateway: Automatic SNAT (no UDR required) NAT Gateway → Internet: Static, deterministic public IP Inbound: NOT supported (outbound only) How it works (step by step) VM initiates an outbound connection (e.g., HTTPS to an API) NAT Gateway intercepts the flow at the subnet boundary Source IP is translated to the NAT Gateway's static public IP The packet is forwarded to the internet Return traffic is automatically tracked and delivered back to the VM No UDRs. No routing tables. No inline devices. It just works. Strengths Massive SNAT scale — no port exhaustion concerns at typical enterprise scale Deterministic outbound IPs — easy to allowlist with external services Zone resilient — survives availability zone failures Subnet scoped — applies to all VMs in the subnet automatically No routing configuration required Limitations ❌ No traffic inspection or filtering ❌ No FQDN or URL policy enforcement ❌ No threat intelligence integration ❌ Cannot restrict which internet destinations are allowed Best Fit Use Cases ✅ Application tiers calling external SaaS APIs ✅ VMs requiring OS updates and patch downloads ✅ CI/CD build agents and pipeline runners ✅ Spoke VNets in hub‑and‑spoke where east‑west goes through firewall, but simple internet egress is acceptable ✅ Dev/test environments Scenario 2: Azure Firewall What is Azure Firewall? Azure Firewall is a cloud‑native, stateful, L3–L7 network security service. When used for outbound egress, it transforms the egress path from a connectivity function into a security enforcement boundary. Unlike NAT Gateway, Azure Firewall inspects every packet, evaluates it against policy, and either allows or denies it based on network rules, application rules, and threat intelligence feeds. KEY Flow are depicted below: VM → UDR: Forces ALL outbound traffic to Firewall Firewall: Evaluates against policy before allowing Firewall → Internet: Only explicitly permitted flows pass All denied flows: Logged and alertable How it works (step by step) VM initiates an outbound connection UDR intercepts the flow and redirects to Azure Firewall's private IP Azure Firewall evaluates the traffic: Network rules (IP/port match) Application rules (FQDN/URL match) Threat intelligence (known malicious IPs/domains) If allowed: traffic is forwarded via Firewall's public IP If denied: traffic is dropped and logged All flows (allowed and denied) are logged to Log Analytics / Sentinel Strengths ✅ Full L3–L7 inspection ✅ FQDN and URL‑based filtering (application rules) ✅ Threat intelligence integration (Microsoft TI feed) ✅ TLS inspection (Premium SKU) ✅ Centralized governance across multiple VNets via Firewall Manager ✅ Rich logging — every allowed and denied flow is recorded ✅ IDPS (Intrusion Detection and Prevention) available in Premium Limitations ❌ Higher cost (hourly + data processing charges) ❌ Requires UDR configuration on each spoke subnet ❌ Adds latency (small but non‑zero) ❌ Requires careful SNAT configuration at scale Best Fit Use Cases ✅ Regulated industries (financial services, healthcare, government) ✅ Any workload where outbound internet is a security boundary ✅ Environments requiring egress allowlisting for compliance ✅ Hub‑and‑spoke architectures with centralized control plane ✅ SOC environments needing outbound flow telemetry Scenario 3: Load Balancer Outbound What is Load Balancer Outbound? Azure Load Balancer outbound rules were historically the primary mechanism for providing SNAT to VMs behind a Standard Load Balancer. While newer patterns (NAT Gateway, Azure Firewall) have largely replaced this approach for new designs, outbound rules remain valid in specific scenarios. Key flows are depicted below: VMs → Load Balancer: Backend pool members get SNAT LB Outbound Rules: Define port allocation per VM ⚠️ Port exhaustion risk at scale ⚠️ No inspection or policy enforcement How it works (step by step) VM in the backend pool initiates an outbound connection Load Balancer applies SNAT using the frontend public IP Ephemeral ports are allocated per VM from a fixed pool Return traffic is tracked and delivered back to the correct VM If port pool is exhausted: connections fail (SNAT exhaustion) Strengths Lower cost than NAT Gateway or Firewall Tightly integrated with existing load‑balanced workloads Familiar operational model for legacy teams Limitations ❌ SNAT port pool is fixed and must be manually managed ❌ Risk of SNAT exhaustion at scale ❌ No traffic inspection ❌ Less flexible than NAT Gateway ❌ Not recommended for new designs Best Fit Use Cases ✅ Existing architectures already built around Azure Load Balancer ✅ Low outbound connection volume workloads ✅ Transitional architectures during modernization to NAT Gateway Decision Framework: Choosing the Right Outbound Pattern Common Pitfalls to Avoid ⚠️ Pitfall 1: Forgetting SNAT scale limits Load Balancer outbound rules allocate a fixed number of ephemeral ports per VM. At scale this exhausts quickly. Use NAT Gateway instead. ⚠️ Pitfall 2: Over‑securing low‑risk workloads Not every workload needs Azure Firewall for outbound. Dev/test and patch traffic are better served by NAT Gateway — simpler, cheaper, faster. ⚠️ Pitfall 3: Mixing outbound models in the same subnet NAT Gateway and Load Balancer outbound rules cannot coexist on the same subnet. NAT Gateway always takes precedence. Plan your subnet boundaries carefully. ⚠️ Pitfall 4: Blocking Azure platform dependencies Many Azure services still use public endpoints (even when Private Link is available). Ensure your outbound policy allows required Azure service tags before enforcing egress controls. ⚠️ Pitfall 5: Relying on platform defaults Default outbound access is retired for new VNets. Do not assume VMs can reach the internet without explicit configuration. Summary and Key Takeaways Scenario Best Choice Why Simple internet egress at scale NAT Gateway Scalable, predictable, no complexity Security boundary for egress Azure Firewall Inspection, FQDN rules, threat intel Legacy load‑balanced workloads Load Balancer Outbound Transitional only Regulated / compliance environments Azure Firewall Audit logs, policy enforcement Dev / test / patch traffic NAT Gateway Low cost, low friction The core principle Private subnets make outbound access intentional. Choose the outbound pattern that matches the risk level of the workload — not the most complex option available. References https://learn.microsoft.com/azure/nat-gateway/nat-overview https://learn.microsoft.com/azure/firewall/overview https://learn.microsoft.com/azure/load-balancer/outbound-rules https://azure.microsoft.com/blog/default-outbound-access-for-vms-in-azure-will-be-retired
How AI Agents Are Turning Threat Intelligence Into Validated Detections
The promise of AI-assisted cybersecurity has long been hampered by a fundamental measurement problem: how do organizations validate whether an AI agent can actually perform the complex, multi-step work that security analysts do every day? Traditional benchmarks test whether models can recall MITRE ATT&CK techniques or classify threat actor tactics, but they miss the harder question—can an agent translate raw threat intelligence into production-ready detection rules that find real attacks?microsoft Microsoft Research has addressed this gap with CTI-REALM (Cyber Threat Intelligence Real World Evaluation and LLM Benchmarking), an open-source benchmark that evaluates AI agents on end-to-end detection engineering workflows. Released in March 2026, CTI-REALM measures whether agents can read threat intelligence reports, explore telemetry schemas, iteratively refine KQL queries, and produce validated Sigma rules and KQL detection logic—exactly the workflow security analysts follow when building detections for platforms like Microsoft Sentinel.microsoft Why Traditional Benchmarks Fall Short Existing cybersecurity AI benchmarks primarily test parametric knowledge—can a model name the technique behind a log entry, or correctly label a tactic from a threat report? While useful, these assessments evaluate isolated skills rather than the operational capability security teams actually need: translating narrative threat intelligence into working detection logic that identifies attacks in production environments.microsoft CTI-REALM fills this gap by measuring three critical dimensions that earlier benchmarks overlook:microsoft Operationalization over recall: Agents must produce working Sigma rules and KQL queries validated against real attack telemetry, not just answer multiple-choice questions about threat actors. Complete workflow evaluation: The benchmark scores intermediate decision quality—CTI report selection, MITRE technique mapping, data source identification, and iterative query refinement—not just final output. Realistic tooling: Agents use the same tools security analysts rely on: CTI repositories, schema explorers, Kusto query engines, and MITRE ATT&CK databases. This granular, checkpoint-based scoring reveals precisely where AI agents struggle in the detection pipeline, helping security leaders understand whether performance gaps stem from comprehension failures, query construction issues, or detection specificity problems.microsoft The Benchmark: Real Threat Intelligence, Real Azure Environments Microsoft curated 37 CTI reports from public sources including Microsoft Security, Datadog Security Labs, Palo Alto Networks, and Splunk, selecting scenarios that could be faithfully simulated in sandboxed environments with telemetry suitable for detection development.microsoft The benchmark spans three Azure-relevant platforms: Linux endpoints: Traditional host-based detection scenarios Azure Kubernetes Service (AKS): Container and orchestration layer attacks Azure cloud infrastructure: Multi-source, APT-style attack chains requiring correlation across identity, resource, and network logs Ground-truth scoring validates detection rules at every workflow stage, from technique identification through final KQL query accuracy.microsoft Key Findings: What Works, What Doesn't Microsoft evaluated multiple frontier AI models on CTI-REALM-50, a subset spanning all three platforms. The results reveal both promise and clear limitations:microsoft Performance drops sharply across platform complexity: Linux endpoint detections scored 0.585, AKS scenarios dropped to 0.517, and Azure cloud infrastructure plummeted to 0.282. This reflects the reality that multi-source correlation across identity logs, Azure Activity, and resource-specific telemetry remains exceptionally difficult for AI agents—precisely the scenario SOC teams working in Microsoft Sentinel face when investigating sophisticated, multi-stage cloud attacks.microsoft More reasoning isn't always better: Within model families, medium reasoning configurations consistently outperformed high reasoning modes, suggesting that overthinking hurts performance in tool-rich, iterative agentic environments.microsoft Structured guidance closes performance gaps: Providing smaller models with human-authored workflow guidance improved threat technique identification and closed approximately one-third of the performance gap to much larger models.microsoft What This Means for Azure Security Operations For security architects and SOC teams working with Microsoft Sentinel, CTI-REALM's findings have immediate practical implications: Traditional Detection Engineering AI-Assisted Detection Engineering Analyst reads threat report manually AI agent parses CTI report and extracts techniques Analyst identifies relevant MITRE techniques Agent maps techniques to data sources automatically Analyst explores schema, writes KQL queries Agent iterates on KQL queries using schema tools Analyst validates detection against test data Agent generates Sigma rule + KQL validated against telemetry Process takes hours to days per report Process completes in minutes with human validation The benchmark demonstrates that AI agents can meaningfully accelerate detection development, particularly for Linux and AKS scenarios where success rates exceed 50%. However, the 28% success rate for Azure cloud infrastructure detections underscores a critical reality: human expertise remains essential for validating complex, multi-source detections before operational deployment.microsoft+1 Security teams should view AI agents as analyst augmentation tools rather than replacements. The checkpoint-based scoring in CTI-REALM helps organizations identify where human review is most critical—typically in cloud correlation logic, detection specificity tuning, and false positive reduction. Responsible Adoption: Human-in-the-Loop Remains Non-Negotiable Microsoft's research reinforces that AI-generated detection rules require validation before production use. Organizations adopting AI-assisted detection workflows should implement structured governance:microsoft Validate AI-generated KQL queries against test datasets before enabling in Sentinel analytics rules Require peer review for detections targeting cloud infrastructure, where AI performance is weakest Benchmark models using CTI-REALM before considering downstream operational use Maintain detection metadata tracking whether rules originated from AI or human analysts to support incident response context The benchmark's open-source availability on the Inspect AI repository enables security teams to test models against their own operational requirements before adoption.microsoft The Path Forward CTI-REALM represents a foundational shift in how the security industry evaluates AI capabilities—moving from knowledge recall to operational competence. For Azure practitioners, this matters because the benchmark's platforms (Linux, AKS, Azure cloud) and output formats (Sigma rules, KQL queries) directly mirror working with Microsoft Sentinel's analytics engine.microsoft As Microsoft continues integrating AI capabilities into Security Copilot and the broader unified SIEM+XDR vision, benchmarks like CTI-REALM provide the measurement framework security leaders need to adopt AI responsibly—understanding both capabilities and limitations before operationalizing agent-assisted workflows. The benchmark is freely available to model developers and security teams. Organizations interested in contributing, benchmarking, or exploring partnership opportunities can access the repository and contact Microsoft Research at msecaimrbenchmarking@microsoft.com.microsoft About the Research: CTI-REALM was developed by Microsoft Research and announced March 20, 2026. The full technical paper is available at CTI-REALM: A new benchmark for end-to-end detection rule generation with AI agents | Microsoft Security Blog
Entra ID Login via Azure Bastion Fails After VM Recreation
However, you may encounter a confusing scenario where: An Entra ID user attempts to sign in to a Windows VM through Azure Bastion The connection appears to succeed in the backend logs The session is disconnected within a second Bastion returns a generic sign-in error to the user At first glance, everything looks correctly configured. Terraform applies cleanly, permissions are in place, and Bastion access is allowed. This blog walks through a real-world troubleshooting journey that exposes a non-obvious Entra ID device registration issue, explains the root cause, and provides a clean resolution. Scenario We manage Azure infrastructure using Terraform, with Entra ID login enabled via the AADLoginForWindows VM extension. Azure Bastion is used to provide secure, inbound‑port‑free access to Windows VMs. After deleting and recreating a Windows VM with the same hostname, Entra ID login through Bastion started failing. Traditional local admin login worked, but Entra ID–based access did not. Key Terraform Configuration The VM was deployed with Entra ID login enabled using Infrastructure as Code: AADLoginForWindows extension Role assignments: Virtual Machine Administrator Login or Virtual Machine User Login Bastion configured for Entra ID authentication From an IaC perspective, nothing was misconfigured. Symptoms Observed The issue manifested in multiple subtle ways: Bastion login using Entra ID fails with a generic error message Backend logs show authentication success Session disconnects immediately after connection establishment Running the following on the VM: dsregcmd /status shows: IsDeviceJoined: NO This explains why Entra ID authentication succeeds initially but instantly fails during session creation. Root Cause Explained When a Windows VM is joined to Microsoft Entra ID, a device object is created in Entra ID, keyed to the VM’s Windows hostname. If the VM is later deleted without removing the device object, and a new VM is recreated using the same hostname, the Entra ID join process fails silently due to a hostname collision. Key points: The old Entra ID device object still exists The new VM cannot complete Entra ID registration Bastion authentication succeeds, but authorization fails immediately The VM therefore disconnects the session This is why backend logs look “successful” even though the user experience is not. Resolution Steps Identify the Stale Device Object Navigate to: Azure Portal → Microsoft Entra ID → Devices → All devices Search for the VM hostname (for example, VM01) Open the device object and note the Object ID Confirm it matches the Object ID referenced in the extension logs. Delete the Stale Device This does not delete the VM or any Azure resources. Only the Entra ID registration is removed. You can delete the device using either method: Azure Portal Select the device Choose Delete Azure CLI az ad device delete --id <ObjectId> Retry the Entra ID Join Restart the VM or restart the AADLoginForWindows extension Wait for the extension to re‑execute Verify the join status: dsregcmd /status Expected output: IsDeviceJoined: YES Retry Bastion login using Entra ID The session should now remain connected and function normally Why This Issue Is Easy to Miss Azure VM deletion does not automatically clean up Entra ID device objects Terraform recreations with identical hostnames are common in non‑prod environments Bastion logs are not explicit about device join failures Authentication succeeds, but authorization fails post‑connection Key Takeaways Deleting a device from Microsoft Entra ID does not impact the VM itself Always check for stale Entra ID device objects when reusing hostnames dsregcmd /status is the fastest way to validate join state AADLoginForWindows extension logs are critical for root cause analysis Bastion disconnections immediately after login often indicate identity‑level issues, not networking problems References Troubleshoot Microsoft Entra ID device registration issues Manage and delete stale Entra ID devices AADLoginForWindows extension documentationEnterprise UAMI Design in Azure: Trust Boundaries and Blast Radius
As organizations move toward secretless authentication models in Azure, Managed Identity has become the preferred approach for enabling secure communication between services. User Assigned Managed Identity (UAMI) in particular offers flexibility that allows identity reuse across multiple compute resources such as: Azure App Service Azure Function Apps Virtual Machines Azure Kubernetes Service While this flexibility is beneficial from an operational perspective, it also introduces architectural considerations that are often overlooked during initial implementation. In enterprise environments where shared infrastructure patterns are common, the way UAMI is designed and assigned can directly influence the effective trust boundary of the deployment. Understanding Identity Scope in Azure Unlike System Assigned Managed Identity, a UAMI exists independently of the compute resource lifecycle and can be attached to multiple services across: Resource Groups Subscriptions Environments This capability allows a single identity to be reused across development, testing, or production services when required. However, identity reuse across multiple logical environments can expand the operational trust boundary of that identity. Any permission granted to the identity is implicitly inherited by all services to which the identity is attached. From an architectural standpoint, this creates a shared authentication surface across isolated deployment environments. High-Level Architecture: Shared Identity Pattern In many enterprise Azure deployments, it is common to observe patterns where: A single UAMI is assigned to multiple App Services The same identity is reused across automation workloads Identities are provisioned centrally and attached dynamically While this simplifies management and avoids identity sprawl, it may also introduce unintended privilege propagation across services. For example: In this architecture: Multiple App Services across environments share the same managed identity. Each compute instance requests an access token from Microsoft Entra ID using Azure Instance Metadata Service (IMDS). The issued token is then used to authenticate against downstream platform services such as: Azure SQL Database Azure Key Vault Azure Storage Because RBAC permissions are assigned to the shared identity rather than the compute instance itself, the effective authentication boundary becomes identity‑scoped instead of environment‑scoped. As a result, any compromised lower‑tier environment such as DEV may obtain an access token capable of accessing production‑level resources if those permissions are assigned to the shared identity. This expands the operational trust boundary across environments and increases the potential blast radius in the event of identity misuse. Blast Radius Considerations Blast radius refers to the potential impact scope of a security or configuration compromise. When a shared UAMI is used across multiple services, the following conditions may increase the blast radius: Design Pattern Potential Risk Single UAMI across environments Cross‑environment access Subscription‑wide RBAC assignment Broad privilege scope Identity used for automation pipelines Lateral movement Shared identity across teams Ownership ambiguity Because Managed Identity authentication relies on Azure Instance Metadata Service (IMDS), any compromised compute resource with access to IMDS may request an access token using the attached identity. This token can then be used to authenticate with downstream Azure services for which the identity has RBAC permissions. Enterprise Design Recommendations: Environment‑Isolated Identity Model To reduce identity blast radius in enterprise deployments, the following architectural principles may be considered: Environment‑Scoped Identity Provision separate UAMIs per environment: UAMI‑DEV UAMI‑UAT UAMI‑PROD Avoid reusing the same identity across isolated lifecycle stages. Resource‑Level RBAC Assignment Prefer assigning RBAC permissions at: Resource Resource Group instead of Subscription scope wherever feasible. Identity Ownership Model Ensure ownership clarity for identities assigned across shared workloads. Identity lifecycle should be aligned with: Application ownership Service ownership Deployment boundary Least Privilege Assignment Assign roles such as: Key Vault Secrets User Storage Blob Data Reader instead of broader roles such as: Contributor Owner Recommended High‑Level Architecture In this architecture: Each App Service instance is attached to an environment‑specific managed identity. RBAC assignments are scoped at the resource or resource group level. Microsoft Entra ID issues tokens independently for each identity. Trust boundaries remain aligned with deployment environments. A compromised DEV compute instance can only obtain a token associated with UAMI‑DEV. Because UAMI‑DEV does not have RBAC permissions for production resources, lateral access to PROD dependencies is prevented. Blast Radius Containment: This design significantly reduces the potential blast radius by ensuring that: Identity compromise remains environment‑scoped. Token issuance does not grant unintended cross‑environment privileges. RBAC permissions align with application ownership boundaries. Authentication trust boundaries match deployment lifecycle boundaries. Conclusion User Assigned Managed Identity offers significant advantages for secretless authentication in Azure environments. However, architectural considerations related to identity reuse and scope of assignment must be evaluated carefully in enterprise deployments. By aligning identity design with trust boundaries and minimizing the blast radius through scoped RBAC and environment isolation, organizations can implement Managed Identity in a way that balances operational efficiency with security governance.Private DNS and Hub–Spoke Networking for Enterprise AI Workloads on Azure
Introduction As organizations deploy enterprise AI platforms on Azure, security requirements increasingly drive the adoption of private-first architectures. Private networking only Centralized firewalls or NVAs Hub–and–spoke virtual network architectures Private Endpoints for all PaaS services While these patterns are well understood individually, their interaction often exposes hidden failure modes, particularly around DNS and name resolution. During a recent production deployment of a private, enterprise-grade AI workload on Azure, several issues surfaced that initially appeared to be platform or service instability. Closer analysis revealed the real cause: gaps in network and DNS design. This post shares a real-world technical walkthrough of the problem, root causes, resolution steps, and key lessons that now form a reusable blueprint for running AI workloads reliably in private Azure environments. Problem Statement The platform was deployed with the following characteristics: Hub and spoke network topology Custom DNS servers running in the hub Firewall / NVA enforcing strict egress controls AI, data, and platform services exposed through Private Endpoints Azure Container Apps using internal load balancer mode Centralized monitoring, secrets, and identity services Despite successful infrastructure deployment, the environment exhibited non-deterministic production issues, including: Container Apps intermittently failing to start or scale AI platform endpoints becoming unreachable from workload subnets Authentication and secret access failures DNS resolution working in some environments but failing in others Terraform deployments stalling or failing unexpectedly Because the symptoms varied across subnets and environments, root cause identification was initially non-trivial. Root Cause Analysis After end-to-end isolation, the issue was not AI services, authentication, or application logic. The core problem was DNS resolution in a private Azure environment. 1. Custom DNS servers were not Azure-aware The hub DNS servers correctly resolved: Corporate domains On‑premises records However, they could not resolve Azure platform names or Private Endpoint FQDNs by default. Azure relies on an internal recursive resolver (168.63.129.16) that must be explicitly integrated when using custom DNS. 2. Missing conditional forwarders for private DNS zones Many Azure services depend on service-specific private DNS zones, such as: privatelink.cognitiveservices.azure.com privatelink.openai.azure.com privatelink.vaultcore.azure.net privatelink.search.windows.net privatelink.blob.core.windows.net Without conditional forwarders pointing to Azure’s internal DNS, queries either: Failed silently, or Resolved to public endpoints that were blocked by firewall rules 3. Container Apps internal DNS requirements were overlooked When Azure Container Apps are deployed with: internal_load_balancer_enabled = true Azure does not automatically create supporting DNS records. The environment generates: A default domain .internal subdomains for internal FQDNs Without explicitly creating: A private DNS zone matching the default domain *, @, and *.internal wildcard records internal service-to-service communication fails. 4. Private DNS zones were not consistently linked Even when DNS zones existed, they were: Spread across multiple subscriptions Linked to some VNets but not others Missing links to DNS server VNets or shared services VNets As a result, name resolution succeeded in one subnet and failed in another, depending on the lookup path. Resolution No application changes were required. Stability was achieved entirely through architectural corrections. ✅ Step 1: Make custom DNS Azure-aware On all custom DNS servers (or NVAs acting as DNS proxies): Configure conditional forwarders for all Azure private DNS zones Forward those queries to: 168.63.129.16 This IP is Azure’s internal recursive resolver and is mandatory for Private Endpoint resolution. ✅ Step 2: Centralize and link private DNS zones A centralized private DNS model was adopted: All private DNS zones hosted in a shared subscription Linked to: Hub VNet All spoke VNets DNS server VNet Any operational or virtual desktop VNets This ensured consistent resolution regardless of workload location. ✅ Step 3: Explicitly handle Container Apps DNS For Container Apps using internal ingress: Create a private DNS zone matching the environment’s default domain Add: * wildcard record @ apex record *.internal wildcard record Point all records to the Container Apps Environment static IP Add a conditional forwarder for the default domain if using custom DNS This step alone resolved multiple internal connectivity issues. ✅ Step 4: Align routing, NSGs, and service tags Firewall, NSG, and route table rules were aligned to: Allow DNS traffic (TCP/UDP 53) Allow Azure service tags such as: AzureCloud CognitiveServices AzureActiveDirectory Storage AzureMonitor Ensure certain subnets (e.g., Container Apps, Application Gateway) retained direct internet access where required by Azure platform services Key Learnings 1. DNS is a Tier‑0 dependency for AI platforms Many AI “service issues” are DNS failures in disguise. DNS must be treated as foundational platform infrastructure. 2. Private Endpoints require Azure DNS integration If you use: Custom DNS ✅ Private Endpoints ✅ Then forwarding to 168.63.129.16 is non‑negotiable. 3. Container Apps internal ingress has hidden DNS requirements Internal Container Apps environments will not function correctly without manually created DNS zones and .internal records. 4. Centralized DNS prevents environment drift Decentralized or subscription-local DNS zones lead to fragile, inconsistent environments. Centralization improves reliability and operability. 5. Validate networking first, then the platform Before escalating issues to service teams: Validate DNS resolution Verify routing Check Private Endpoint connectivity In many cases, the perceived “platform issue” disappears. Quick Production Validation Checklist Before go-live, always validate: ✅ Private FQDNs resolve to private IPs from all required VNets ✅ UDR/NSG rules allow required Azure service traffic ✅ Managed identities can access all dependent resources ✅ AI portal user workflows succeed (evaluations, agents, etc.) ✅ terraform plan shows only intended changes Conclusion Running private, enterprise-grade AI workloads on Azure is absolutely achievable—but it requires intentional DNS and networking design. By: Making custom DNS Azure-aware Centralizing private DNS zones Explicitly handling Container Apps DNS Aligning routing and firewall rules an unstable environment was transformed into a repeatable, production-ready platform pattern. If you are building AI solutions on Azure with Private Endpoints and hub–spoke networking, getting DNS right early will save weeks of troubleshooting later.
