Azure Infrastructure Blog articles

CHERIoT-Ibex: Closing the door on memory safety vulnerabilities with hardware-enforced protection

kunyanliu — Sat, 09 May 2026 05:08:11 GMT

Memory safety vulnerabilities—largely arising from widely used programming languages such as C and C++—remain a leading cause of exploitable software defects across systems, from embedded devices to cloud-scale infrastructure. In simple terms, memory safety ensures that software accesses only the data it is intended to use; when this protection fails, attackers can exploit these defects to gain control of devices or disrupt critical services. 

Industry data shows that about 70 percent of the vulnerabilities Microsoft assigns as Common Vulnerabilities and Exposures (CVE) each year are memory safety issues, highlighting how frequently these software defects translate into real-world security risk (CISA – The Urgent Need for Memory Safety in Software Products). Hardware-enforced protections such as CHERIoT-Ibex can help eliminate these vulnerabilities at their source, reducing the likelihood that low-level software flaws can be exploited to compromise devices or disrupt workloads, supporting more trustworthy infrastructure by design. 

An open and certified foundation for memory-safe embedded systems

CHERIoT-Ibex is the first open-source production-quality implementation of the CHERIoT instruction set architecture and among the first cores certified by the CHERI Alliance (CHERI Alliance – CHERIoT). CHERIoT is an extension of the CHERI (Capability Hardware Enhanced RISC Instructions) instruction set, with a focus on embedded and Internet of Things (IoT) applications. Ibex is an open‑source 32‑bit RISC‑V core developed by LowRISC. CHERIoT‑Ibex builds on Ibex by including CHERIoT capability extensions to provide hardware‑enforced memory safety and fine‑grained compartmentalization. It is the result of a close partnership between Microsoft Research and Azure Hardware Systems & Infrastructure, combining advanced research innovation with industry-leading silicon IP development expertise. 

In 2023, Microsoft open-sourced the CHERIoT Platform to bring hardware-enforced memory safety to embedded systems, including an instruction set architecture, toolchain, real-time operating system, and the RTL implementation of the CHERIoT-Ibex core. The CHERI Alliance certification recognizes its ability to provide spatial and temporal memory safety, fine-grained compartmentalization, and compatibility with the broader CHERI ecosystem. Critically, CHERIoT-Ibex achieves these security guarantees with power and area efficiency comparable to low-cost microcontrollers, demonstrating that security doesn’t have to come at a premium. 

Why memory safety remains a foundational security challenge

Traditional embedded and microcontroller-class designs rely on software hardening and coarse-grained hardware protections that struggle to prevent attacks such as buffer overflows and use-after-free vulnerabilities, often adding complexity while still leaving gaps in protection. 

Consider a controller that runs privileged firmware responsible for device initialization, telemetry, and system health monitoring, while also hosting networking functionality exposed to external inputs. A memory-safe vulnerability in the networking stack could allow attackers to execute unauthorized code within the firmware environment, potentially affecting other critical services on the device. In tightly integrated systems, these failures can propagate beyond a single component, increasing overall risk.

Constraining failures with hardware-enforced isolation

CHERIoT-Ibex enables hardware-enforced isolation between these components, helping ensure that even if the networking stack is compromised, its ability to impact system initialization or telemetry functions remains constrained. By limiting the blast radius of software failures, CHERIoT-Ibex supports a system-level approach to security rather than relying on individual components to defend themselves in isolation.

Advancing memory-safe infrastructure by design

CHERIoT-Ibex’s certification by the CHERI Alliance marks an important milestone for open-source memory-safe solutions. It validates that strong security guarantees can coexist with efficiency and transparency, reflecting Microsoft’s broader silicon-to-systems strategy of embedding security into the foundational hardware infrastructure.

Explore and engage with the open-source CHERIoT ecosystem by visiting the CHERIoT Platform and the CHERIoT-Ibex GitHub repository (microsoft/cheriot-ibex). The repositories enable developers and researchers to experiment with, contribute to, and build on memory-safe hardware and software foundations. 

Safely Migrating Terraform Managed Disks on Azure Using Stable Keys and Copilot

shwetayadav — Fri, 08 May 2026 16:02:01 GMT

The Root Cause: Index-Based "for_each" Keys:

Many Terraform modules flatten VM and disk definitions into a list and use the list index as the for_each key:

for_each = { for index, sp in local.managed_disks : index => sp }

This pattern looks harmless, but the index is not stable:

Adding a disk to one VM shifts downstream indices
Reordering environment JSON changes flatten order
Terraform treats shifted indices as new resources

The result: Terraform plans to destroy and recreate all affected managed disks—even though nothing changed in Azure.

Why This Is Especially Risky on Azure:

Azure managed disks are often:

Attached to stateful application tiers
Used for databases, middleware, or batch workloads
Deployed across zones for resiliency

A forced disk replacement can mean:

Data loss
Extended outages
Failed change windows

This makes state stability a first-class design concern—not an implementation detail.

The Stable Key Pattern:

The fix is conceptually simple: use a domain-stable identifier for each disk.

A proven pattern is:

"${sp.vm}-${sp.data_disk.lun}"

This key is:

Deterministic
Independent of ordering
Human-readable
Stable across environments

Example:

VM	LUN	Stable Key
vm1	0	vm1-0
vm1	1	vm1-1
vm2	0	vm2-0

Once applied, adding a new disk results in exactly one new resource, with zero churn.

The Migration Challenge: Terraform State:

Changing for_each keys alone is not enough.

Terraform tracks resources by their state address, not by Azure resource ID. When keys change, Terraform believes:

“The old disks were deleted, and new ones must be created.”

To prevent this, we must move the state, not recreate the resource.

That is where terraform state mv comes in.

Automating the Migration with GitHub Copilot Skills:

To remove risk and human error, the team created a reusable Copilot skill for managed disk key migration.

What the Skill Does:

Inspects Terraform modules for index-based for_each
Reads environment JSON files (such as ALZ variable abstractions)
Reconstructs the exact flatten order used by Terraform
Generates precise terraform state mv commands

This ensures:

No guessing
No manual address mapping
No production surprises

The skill is stored directly inside the repository under .github/skills, making it:

Discoverable
Versioned
Shareable across teams

Example: Generating State Move Commands:

Based on environment JSON, Copilot can generate commands like:

terraform state mv \

'module.managed_disk_windowsvm_app["0"]' \

'module.managed_disk_windowsvm_app["vm1-0"]'

This is repeated deterministically for every existing disk—before any plan or apply.

Recommended Migration Workflow:

Confirm clean state
- terraform plan shows no pending changes
Update the module
- Replace index-based keys with stable keys
Back up the state
- Especially critical with remote backends (Azure Storage)
Run terraform state mv
- Only after terraform init is connected to the correct backend
Re-run plan
- Existing disks should show no changes
Add new disks safely
- Terraform creates only the new disk

CI/CD and Remote Backend Considerations:

A critical finding from this migration:

terraform state mv always modifies the currently initialized backend.

In pipeline-driven environments:

Ensure the correct environment is initialized
Run migrations once per environment
Never merge stable-key code before migrating all environments

Failing to align code and state can cause disk destruction in production.

Key Takeaways:

Index-based for_each keys are unsafe for long-lived Azure disks
Stable keys such as vm-lun eliminate accidental resource churn
State migration is mandatory—not optional
Copilot skills are powerful for institutionalizing safe patterns
Small Terraform design choices can have enterprise-scale impact

Closing Thoughts:

This pattern is broadly applicable beyond disks—to NICs, extensions, and any resource where identity must outlive ordering.

By combining:

Stable Terraform design
State-aware migrations
GitHub Copilot automation

Teams can make infrastructure changes boring again—and that is the ultimate reliability goal.

Building Secure AI Platforms in Banking Using Azure Enterprise Architecture

divyanshi_varshney — Thu, 07 May 2026 16:03:01 GMT

1. Introduction: AI in Banking Is Not Just a Model Problem

Modern banking institutions are no longer asking “Can we use AI?”
The real question is:
“Can we use AI without violating regulatory, security, and data residency constraints?”

Unlike public AI applications, banking systems must ensure:

No public internet exposure
Strict identity-based access control
End-to-end auditability
Data residency compliance
Fully controlled inference pipelines

👉 In enterprise environments, AI success is driven by secure infrastructure—not just model accuracy.

2. Core Design Principle: Controlled Intelligence System

Every AI request must follow a security-enforced execution pipeline:

User Request ↓ Secure Edge (Application Gateway + WAF) ↓ API Governance Layer (API Management - Internal Mode) ↓ AI Orchestration Layer (AKS / App Services) ↓ Retrieval + Policy Layer (RAG + Guardrails) ↓ Private AI Services (Azure OpenAI) ↓ Observability Layer (AMPLS) ↓ Final Response

Key Insight:
This is not just an architecture—it is a controlled and auditable execution model.

3. Azure Enterprise AI Architecture (Production-Ready Pattern)

A real-world architecture used in banking environments:

4. Private Connectivity Model (Critical for Compliance)

Key components:

Private Endpoints → Secure PaaS isolation
Private DNS Zones → Controlled name resolution
VNet Integration → Internal service communication
Azure Firewall → Traffic inspection and control

⚠️ Common Production Failure:

AKS pods fail to resolve Azure OpenAI private endpoint
Root cause:
- Missing Private DNS links
- Incorrect VNet configuration

👉 This is one of the most frequent failures in enterprise AI deployments.

“Debugging Private Endpoint Failures”

Include:

nslookup behavior in AKS
DNS zone linking check
VNet integration validation
UDR / Firewall inspection

5. Identity-First Security Model (No Secrets Architecture)

Modern banking architectures eliminate static credentials entirely.

Authentication Flow:

AKS Workload → Managed Identity → Azure AD → Azure Services

Key Principle:
👉 Identity is the new security perimeter.

Benefits:

No API keys or secrets
Simplified access management
RBAC-based governance
Fully auditable access

6. Secure AI Inference Pipeline

A production AI request flow:

def process_request(user_request): # 1. Authenticate user via Azure AD identity = authenticate_aad(user_request.token) if not identity or not identity.is_valid: return "ACCESS_DENIED" # 2. Enforce rate limiting per identity if not rate_limit(identity): return "RATE_LIMIT_EXCEEDED" # 3. Apply prompt security guardrails (injection protection) safe_prompt = apply_prompt_guardrails(user_request.prompt) # 4. Content safety filtering (PII / harmful content detection) if not content_filter(safe_prompt): return "CONTENT_BLOCKED" # 5. Retrieve secure RAG context context = retrieve_rag_context( query=safe_prompt, secure_mode=True ) # 6. Build final prompt final_prompt = merge_prompt_and_context(safe_prompt, context) # 7. Call Azure OpenAI with circuit breaker protection response = circuit_breaker( lambda: call_openai( prompt=final_prompt, identity=ManagedIdentity() ) ) # 8. Validate and sanitize model output validated_output = sanitize(response) # 9. Log everything for audit + compliance (AMPLS / SIEM) log_to_ampls( identity=identity, request=user_request, response=validated_output ) return validated_output

Security controls include:

Prompt injection filtering
Context grounding (RAG)
Output sanitization
Full audit logging

7. RAG Architecture: Enterprise AI Backbone

User Query → Embedding Model → Azure AI Search (Vector Store) → Context Retrieval → Azure OpenAI → Final Response

Why RAG is preferred in banking:

No model retraining required
Controlled data exposure
Easier compliance validation
Real-time knowledge updates

In banking systems, retrieval is not just about relevance—it is about controlled disclosure of sensitive context

8. Observability with AMPLS (A Critical Yet Overlooked Layer)

AI telemetry flows through:

Azure Services → Private Link → AMPLS → Log Analytics / App Insights

Why this matters: Logs may contain:

Sensitive financial data
PII
Prompt inputs

👉 AMPLS ensures telemetry remains private and compliant.

9. Regulatory Mapping: Banking Requirements to Azure Capabilities

Requirement	Azure Implementation
No public exposure	Private Endpoints
Identity-based security	Azure AD + Managed Identity
Audit compliance	Log Analytics + AMPLS
Data protection	Customer-Managed Keys (CMK)
Network isolation	VNet + Firewall
Access governance	RBAC + PIM

10. Real-World Production Challenges

Common failure points in enterprise AI systems:

DNS Misconfiguration – Private endpoints fail resolution
Latency Chains – Excessive service hops
OpenAI Rate Limits – High enterprise load
Identity Propagation Issues – Cross-subscription failures
Observability Gaps – Missing distributed tracing

11. Enterprise Architecture Best Practices

Design with zero-trust principles
Treat AI as a distributed system, not a single component
Centralize governance using API Management
Never expose AI services publicly
Use identity everywhere—no secrets
Separate:
- Control Plane (governance)
- Data Plane (inference execution)

12. Azure Service Mapping (Quick Reference)

Layer	Azure Services
Edge Security	Application Gateway (WAF)
API Layer	API Management
Compute	AKS / App Services
AI Services	Azure OpenAI
Retrieval	Azure AI Search
Data	Azure Storage / SQL
Identity	Azure AD + Managed Identity
Networking	Private Link + VNet
Observability	AMPLS + Log Analytics

13. Common Failure Patterns

Issue	Root Cause
AI endpoint unreachable	DNS / Private endpoint misconfig
Data leakage risk	Missing prompt filtering
High latency	Over-layered architecture
Unauthorized access	Identity misconfiguration
Poor response quality	Weak RAG implementation

14. Final Thought

In enterprise banking AI systems:

Models are replaceable. Architecture is not.

The real challenge is designing a system where AI is:

Secure
Controlled
Observable
Fully compliant

How Validation‑Driven Terraform Made Our Azure Function Deployments Predictable

AkshitaBajpai — Thu, 07 May 2026 05:12:06 GMT

When Terraform deploys Azure Functions, the most expensive failures are rarely “syntax” problems. They’re environmental mismatches discovered too late—during terraform apply, after approvals, after a change window starts, and often after multiple teams are already watching the pipeline.

After a few painful production-grade rollouts, we shifted to a validation-driven approach: instead of letting Azure reject misconfigurations at apply time, we fail fast at PR/plan time with clear messages that engineers can fix immediately.

What we mean by “validation‑driven”

Validation-driven Terraform uses three guardrails together:

PR checks: formatting, linting, security scanning, module contract tests
Pre-flight checks: quick Azure sanity checks (provider registration, storage prerequisites, RBAC basics)
Terraform-native validations: input validations + preconditions that stop invalid configurations before they reach Azure

The idea is simple: apply should be boring. If something is going to fail, it should fail earlier with better errors.

Azure Functions as the example: where failures actually happen

Azure Functions bring a few recurring “gotchas” that tend to show up late:

mismatch between plan/SKU and features (e.g., VNET integration expectations)
missing or inaccessible storage account settings
unsupported/incorrect runtime stack/version for a chosen hosting model/region/policy
inconsistent app settings required by your org platform standards

We converted these into guardrails with minimal code and clearer pipeline signals.

Case study 1: Wrong plan SKU causing runtime capability failures

Problem

A team deployed a Function App expecting network integration behavior, but the selected plan/SKU didn’t align with what the workload required. The pipeline failed late, after approvals, and the rollback discussion took longer than the fix.

What we changed

We added a small validation/precondition rule: if a team enables a capability that requires a certain class of plan, Terraform fails early with a targeted message.

Outcome

Failure moved from apply-time → plan-time
2+ hours saved per failed deployment cycle
Zero repeat incidents for that class of issue

Case study 2: Missing storage configuration blocking deployments

Problem

Function Apps depend heavily on storage configuration. We saw intermittent failures when storage settings pointed to deleted/incorrect resources, or when access expectations didn’t match reality.

What we changed

We introduced a pre-flight check step in Azure DevOps: verify storage existence/access and fail fast before plan/apply.

Outcome

Deployments stopped failing mid-run
Fewer “investigation loops” across teams
Reduced incident noise (the pipeline became self-explanatory)

Case study 3: Unsupported runtime version (region/org guardrails mismatch)

Problem

Engineers selected a runtime stack/version that was valid in isolation, but not aligned with platform support or readiness in the target environment. Failures appeared in apply or after release.

What we changed

We centralized an “allowed runtime list” (per org standards) and validated runtime inputs at plan time.

Outcome

Plan failed fast with clear explanation
No redeploy cycles
Better compliance posture (standards became enforceable code)

Why this matters (beyond “fewer red pipelines”)

Validation-driven Terraform improved more than deployment success rate:

Developer experience: errors became precise and actionable
Operational safety: fewer emergency approvals and late-night fixes
Standardization: platform rules stopped living in tribal knowledge and wikis
Manager-visible impact: less delivery friction, fewer escalations, faster releases

The best part: this wasn’t achieved by writing massive frameworks. It was achieved by adding small, high-leverage validations in the right places.

Minimal code approach (what we actually used)

We intentionally kept Terraform code small:

A few input validations (environment, runtime, naming contracts)
A few preconditions (must-have settings and plan constraints)
A light Azure DevOps pre-flight step for checks Terraform can’t reliably infer (like “does this dependency exist and is it accessible?”)

This way, the module stays readable, and the pipeline remains fast.

Conclusion

In one of our Azure Function platforms, repeated deployment failures were not caused by bugs in application code or gaps in Terraform itself—they were caused by discovering platform constraints too late. Each failed terraform apply triggered rework, additional approvals, and unnecessary operational noise across teams.

By introducing a validation‑driven approach—combining Terraform input validations, targeted preconditions, and lightweight Azure DevOps pre‑flight checks—we moved failure discovery to the right place: pull requests and plan stages. Azure Function‑specific issues such as incorrect plan capabilities, unsupported runtimes, and missing storage prerequisites were surfaced early, with clear, actionable messages.

If your Azure DevOps pipelines still use terraform apply as a discovery mechanism, validation is not an optimization—it’s a foundational platform capability.

🚀 Modernizing Azure Landing Zone Deployments: From Terraform Scripts to GHCP-Driven Engineering

gurkirat — Wed, 06 May 2026 06:05:20 GMT

The code nobody wants to write anymore

Most Azure Landing Zone (ALZ) deployments we see today still follow the same pattern they did a few years ago. Engineers manually write Terraform modules, copy networking blocks from older repos, tweak management group hierarchies, and wire pipelines step by step.

It works. But it’s slow, repetitive, and heavily dependent on individual expertise.

The real problem isn’t complexity. It’s friction.

Every new Landing Zone starts with:

Rebuilding Terraform structures
Rewriting GitHub Actions pipelines
Re-validating OIDC authentication
Re-implementing policy assignments

And despite all that effort, most implementations still look… slightly different.

This is exactly where GitHub Copilot (GHCP) changes the game.

Instead of writing infrastructure line by line, we now describe intent — and let AI generate the implementation.

From Infrastructure as Code → Infrastructure by Prompt

The biggest shift isn’t Terraform. It’s how we arrive at Terraform.

Before:

We now start with

That prompt doesn’t just generate code. It generates:

Architecture decisions
Module structure
Naming conventions
Deployment patterns

The engineer moves from author → reviewer.

How GHCP actually accelerates ALZ deployment

This isn’t about autocomplete. It’s about end-to-end acceleration.

Let’s break it down the way it actually happens in real projects.

1. Designing the Landing Zone (in minutes, not days)

The traditional design phase involves whiteboarding, documentation, and multiple iterations.

With GHCP, we start with:

🔹 Prompt

✅ Output from GHCP

What comes back:

A full MG hierarchy
Suggested subscription model
Terraform module breakdown
Governance baseline

You’re not starting from scratch anymore. You’re starting from a structured draft.

2. Generating Terraform Modules instead of writing them

Instead of building modules manually:

🔹 Prompt

✅ Output

Folder structure
Input/output variables
Reusable module patterns

What used to take hours of setup becomes a review exercise.

3. Networking without copy-paste debt

Networking is where most ALZ implementations diverge.

Instead of digging through old repos:

🔹 Prompt

✅ Output

You get:

Clean networking configuration
Hub definition
Routing setup

More importantly, it’s consistent every time.

4. OIDC setup without documentation rabbit holes

OIDC is well documented. But stitching it together across Azure + GitHub still takes time.

🔹 Prompt

✅ Output

Exact CLI commands
Correct subject format
Proper RBAC scope

No guesswork. No trial-and-error.

5. GitHub Actions workflows generated in seconds

Pipeline creation is one of the most repetitive tasks in ALZ setups.

🔹 Prompt

✅ Output

Fully functional workflow
Correct permissions (id-token: write)
Environment-based deployment

What used to be boilerplate is now instant scaffolding.

6. Policy as Code without digging through definitions

Policy assignment is often delayed because it’s tedious.

🔹 Prompt

✅ Output

Ready-to-use policy assignments
Initiative structure
Correct scopes

Governance is no longer an afterthought.

What actually changes in real projects

This isn’t theoretical. The impact shows up immediately.

Area	Before GHCP	After GHCP
Design	Whiteboarding + docs	Prompt-driven
Terraform	Manual authoring	AI-generated + reviewed
Pipelines	Built from scratch	Generated instantly
OIDC setup	Trial & error	Prompt-guided
Consistency	Varies per engineer	Standardized

The biggest gain isn’t speed. It’s consistency at scale.

The new skill: Prompt Engineering for Cloud

GHCP doesn’t remove the need for expertise. It changes where that expertise is applied.

Bad prompt:

Good prompt:

The quality of output is directly tied to the quality of the prompt.

What still matters (and always will)

Even with GHCP, some things don’t change:

You still validate architecture decisions
You still review Terraform before applying
You still design RBAC carefully
You still enforce governance

GHCP accelerates the how, not the why.

Where this is going next

We’re already seeing GHCP extend beyond:

Subscription vending
Multi-region deployments
Drift detection
Cost governance

The pattern is clear:

Infrastructure is no longer written first. It’s generated, then refined.

Wrapping up

If you do one thing after reading this, try this:

Open your repo and prompt:

Look at the output.

That’s not a shortcut. That’s the new baseline.

GHCP doesn’t replace Terraform.
It removes the friction around writing it.

And in large-scale Azure environments, that’s the difference between:

Moving fast
And moving consistently at scale

Operationalizing Responsible AI in Microsoft Foundry within Enterprise Network Boundaries

Shruti9162 — Wed, 06 May 2026 05:23:44 GMT

Strategic Overview

Deploying Microsoft Foundry within a VNet-integrated landing zone requires a thoughtful balance between innovation and enterprise-grade security, especially in highly regulated industries like banking. This architecture enforces Responsible AI (RAI) principles and robust content safety controls while aligning with stringent security and compliance requirements.

By adopting a dual-stream design - comprising an AI Platform Layer and a Data Integration Layer, you can decouple model orchestration from data ingestion, enabling flexibility and scalability. Leveraging private networking constructs such as VNets, subnets, NSGs, and controlled routing ensures that all AI workloads operate within secure boundaries, while seamless integration with services like Azure AI Search, Azure Cosmos DB, Azure SQL Database, and Azure Document Intelligence enhances data accessibility and intelligence.

Event-driven ingestion patterns powered by Azure Data Factory and Azure Event Grid further enable real-time responsiveness. At the same time, real-world constraints - such as IP range allowlisting for Microsoft Foundry and private networking limitations—must be carefully accounted for. Ultimately, this approach ensures a secure, compliant, and scalable foundation for enterprise AI adoption.

Below are the pointers that this blog focuses:

Deploy Azure Microsoft Foundry in a VNet-integrated landing zone
Enforce Responsible AI (RAI) policies and content safety controls
Align AI architecture with enterprise (banking) security requirements
Implement a dual-stream architecture:
- AI Platform Layer
- Data Integration Layer
Use private networking with VNet, subnets, NSGs, and routing
Integrate with Azure AI Search, Cosmos DB, SQL, and Document Intelligence
Enable event-driven ingestion using Azure Data Factory and Event Grid
Account for real-world constraints:
- IP range allowlisting for Microsoft Foundry
- Private networking limitations
Design for secure, compliant, and scalable AI consumption

Problem Statement

Operationalizing Responsible AI in enterprise environments requires more than defining policies.

Key challenges include:

Translating Responsible AI principles into enforceable platform controls
Deploying AI services within private, enterprise-grade networks
Managing network constraints and service limitations
Ensuring consistent integration across:
- AI services
- Data pipelines
- Application layers

Without a structured approach, AI platforms risk being non-compliant, insecure, or difficult to scale.

Goals

Design an AI landing zone that:

Enforces Responsible AI at the platform level
Enables secure model deployment and access
Operates fully within private network boundaries
Integrates AI and Data services seamlessly
Provides governed and controlled AI consumption

Architecture Overview

Structure the platform into four layers:

Network Layer → VNet, subnets, NSGs, routing
AI Platform Layer → Microsoft Foundry, models, RAI policies
Data Layer → ADF, SHIR, Event Grid
Application Layer → Function Apps, Web Apps

Microsoft Foundry Setup in Enterprise Context

Set up Azure Microsoft Foundry as the core AI platform layer.

Key steps:

Create Microsoft Foundry projects to isolate use cases
Deploy models within controlled project boundaries
Restrict access using:
- VNet integration
- Private endpoints
Disable public access wherever possible
Integrate with supporting services:
- Azure AI Search (retrieval)
- Cosmos DB / SQL (data storage)

Design Principle

Treat AI services as governed platform components, not standalone resources.

Responsible AI Implementation

1. RAI Policies

Define policies at the model interaction layer
Configure controls for:
- Output moderation
- Prompt handling
Align policies with organizational compliance requirements

2. Content Safety

Integrate content safety as a mandatory runtime layer
Ensure all model outputs pass through filtering before reaching applications

Content Safety Flow

3. Model Governance

Control model deployment via Microsoft Foundry
Restrict direct access to models
Route all interactions through:
- Function Apps
- API layers

Handling VNet-Integrated Deployment Challenges

Enterprise deployments introduce constraints that must be addressed early.

Challenge 1: Microsoft Foundry VNet Integration

Microsoft Foundry requires careful network planning
Standard enterprise patterns may not work without validation

Challenge 2: IP Range Constraints

When designing the VNet:

10.x.x.x range
→ Not GA for all Azure regions by default
Requires:
→ Allowlisting via Microsoft Product Group
Supported ranges (commonly observed):
- 172.x.x.x
- 192.x.x.x

Recommended Approach

Validate supported IP ranges before finalizing network design
Avoid assuming default enterprise CIDR blocks will work
Plan subnets specifically for AI workloads

Challenge 3: Platform Constraints

AI services may behave differently compared to traditional PaaS services
Validate:
- Private endpoint compatibility
- Service integration within VNet

Challenge 4: Security vs Accessibility

Private deployments improve security but add complexity
Address this by:
- Providing controlled access paths
- Using jump hosts or secure access mechanisms

Key Design Considerations

Treat networking as a core dependency for AI platforms
Enforce Responsible AI across:
- Model layer
- Platform layer
- Runtime layer
Use layered security architecture:
- Network isolation
- Policy enforcement
- Content filtering
Validate constraints early to avoid redesign

Best Practices

Plan IP addressing specifically for AI workloads
Use private endpoints and VNet integration by default
Centralize model access through application layers
Apply Responsible AI controls as mandatory, not optional
Design AI platforms with governance built-in from the start

Conclusion

Operationalizing Responsible AI in Microsoft Azure goes beyond defining policies—it demands tight alignment across AI services, infrastructure, networking, and governance controls. A well-architected AI landing zone provides the foundation for securely deploying models, enforcing content filtering on outputs, and ensuring that access remains strictly within enterprise-defined boundaries.

AI services
Infrastructure
Networking
Governance controls

A well-designed AI landing zone ensures that:

Models are deployed securely
Outputs are governed and filtered
Access is controlled within enterprise boundaries

Responsible AI is not just a policy—it is an architectural outcome driven by platform design, network constraints, and enforcement mechanisms.

This holistic approach transforms Responsible AI from a conceptual guideline into a practical, enforceable outcome of system design. Crucially, early architectural decisions—particularly those related to networking, private access, and service compatibility—have a lasting impact on how effectively Responsible AI can be scaled across the organization.

Deploying Azure Resources with Managed HSM Keys Using Bicep

Roslin_Nivetha — Tue, 05 May 2026 09:30:46 GMT

Architecture Overview

The deployment includes:

Managed HSM instance
Key creation inside HSM
User-assigned managed identity / service principal
Role assignments for key access
Azure resource (e.g., Storage / Databricks / Disk) using CMK

Flow:

Create Managed HSM
Create encryption key
Assign permissions
Deploy resource with CMK reference

Prerequisites

Before starting, ensure:

Azure subscription with proper permissions
Access to create Managed HSM
Knowledge of RBAC vs access policies
Bicep CLI installed

Step 1: Deploy Managed HSM

Managed HSM is different from regular Key Vault:

Uses RBAC only (no access policies)
Requires security domain initialization

Bicep snippet:

resource managedHsm 'Microsoft.KeyVault/managedHSMs@2023-02-01' = {

name: hsmName

location: location

sku: {

name: 'Standard_B1'

family: 'B'

}

properties: {

tenantId: tenant().tenantId

initialAdminObjectIds: [

adminObjectId

]

}

Step 2: Create Key in Managed HSM

resource key 'Microsoft.KeyVault/managedHSMs/keys@2023-02-01' = {

name: '${managedHsm.name}/cmk-key'

properties: {

kty: 'RSA-HSM'

keySize: 2048

}

Step 3: Assign Permissions

Since Managed HSM uses RBAC, assign roles like:

Managed HSM Crypto User
Managed HSM Crypto Officer

resource roleAssignment 'Microsoft.Authorization/roleAssignments@2022-04-01' = {

name: guid(resourceGroup().id, principalId, roleDefinitionId)

properties: {

principalId: principalId

roleDefinitionId: roleDefinitionId

scope: managedHsm

}

Step 4: Configure Resource with CMK

Example: Storage Account encryption

resource storage 'Microsoft.Storage/storageAccounts@2023-01-01' = {

name: storageName

location: location

kind: 'StorageV2'

sku: {

name: 'Standard_LRS'

}

properties: {

encryption: {

keySource: 'Microsoft.Keyvault'

keyvaultproperties: {

keyname: key.name

keyvaulturi: managedHsm.properties.hsmUri

}

Common Challenges

1. Permission Issues

Resource identity must have access to HSM key
Missing role → deployment failure

2. Key Rotation Impact

When keys are rotated:

Resource may not automatically pick latest version
You may need to redeploy or update configuration

3. Deployment Errors

Typical issue:

Storage/Databricks cannot access HSM key

Fix:

Ensure correct RBAC role assignment
Validate principal ID used during deployment

Key Rotation Strategy

Managed HSM supports:

Manual rotation
Rotation policies

Best practice:

Use version-less key URI if supported
Automate redeployment pipeline

When to Use Managed HSM vs Key Vault

Feature	Managed HSM	Key Vault
FIPS Level	Level 3	Level 2
Multi-tenant isolation	No (dedicated)	Yes
RBAC only	Yes	Optional
Cost	Higher	Lower

Conclusion

Using Managed HSM with Bicep enables:

Stronger security with hardware-backed keys
Full automation via Infrastructure as Code
Enterprise-grade compliance

However, it requires careful handling of:

RBAC permissions
Key rotation
Resource integration

Building an AI Agent for Azure Infrastructure Validation

ranjsharma — Tue, 05 May 2026 06:05:23 GMT

1. Introduction

Infrastructure consistency is critical in large-scale Azure environments, especially in migration programs and DevOps-driven deployments. While Infrastructure as Code (IaC) using Terraform improves reproducibility, it does not fully eliminate:

Manual errors in design specifications
Drift between Terraform and deployed resources
Misalignment between approved design (Excel/architecture docs) and deployed state

To address this, we propose building an AI-powered Infrastructure Validation Agent that continuously validates and reconciles:

Excel (Source of Truth)
Terraform (.tf files)
Azure Deployed Resources

This blog explains the architecture, implementation, validation logic, and real-world applicability of such an agent.

2. Problem Statement

In enterprise environments, infrastructure data flows through multiple stages:

Source	Purpose
Excel / Design Sheets	Approved architecture specifications
Terraform	Infrastructure as Code implementation
Azure Portal	Actual deployed infrastructure

3.Common Challenges

Configuration mismatches across stages
Drift due to manual portal changes
Incorrect SKU, region, or configuration deployment
Lack of automated validation before and after deployment

The absence of unified validation leads to compliance risks, deployment errors, and operational inefficiencies.

4. Solution Overview

The proposed solution is an AI-powered validation agent that:

Ingests Excel as configuration input
Parses Terraform configurations
Fetches deployed resource details from Azure

5. Architecture Overview

High-Level Architecture Components

1. Input Layer
  - Excel file (configuration source)
2. Processing Layer
  - Terraform Parser
  - Azure Resource Fetcher
  - AI-based Validator (optional reasoning layer)
3. Comparison Engine
  - Schema-based comparison
  - Drift detection logic
4. Output Layer
  - Validation report (JSON / Excel / HTML)
5. Hosting
  - Azure Function App
6. Optional Enhancements
  - Azure AI Search for semantic matching and reasoning

6. Agent Design (Modular Components)

Module	Description
Excel Reader	Reads and standardizes input
Terraform Parser	Extracts resource configuration
Azure Fetcher	Queries deployed resources
Comparator Engine	Identifies mismatches
AI Validator	Enhances validation and recommendations
Report Generator	Produces actionable outputs

7. Agent Design
Step 1: Read Excel Input

import pandas as pd

ef read_excel(file_path):

df = pd.read_excel(file_path)

df.columns = df.columns.str.strip()

return df

excel_df = read_excel("infra_config.xlsx")

print(excel_df.head())

Step 2:Parse Terraform Files

import hcl2

def parse_terraform(file_path):

with open(file_path, 'r') as file:

data = hcl2.load(file)

resources = []

for resource_type in data.get('resource', []):

for rtype, instances in resource_type.items():

for name, config in instances.items():

resource = {

"resource_type": rtype,

"resource_name": name,

"config": config

}

resources.append(resource)

return resources

tf_resources = parse_terraform("main.tf")

print(tf_resources)

Step 3:Parse Terraform Files

from azure.identity import DefaultAzureCredential

from azure.mgmt.resource import ResourceManagementClient

credential = DefaultAzureCredential()

subscription_id = "your-subscription-id"

resource_client = ResourceManagementClient(credential, subscription_id)

def fetch_azure_resources():

resources = []

for resource in resource_client.resources.list():

res = {

"name": resource.name,

"type": resource.type,

"location": resource.location,

"id": resource.id

}

resources.append(res)

return resources

azure_resources = fetch_azure_resources()

print(azure_resources)

Step 4:Normalize Data

def normalize_excel(df):

return df.to_dict(orient='records')

def normalize_tf(tf_resources):

normalized = []

for res in tf_resources:

normalized.append({

"resource_name": res["resource_name"],

"resource_type": res["resource_type"],

"config": res["config"]

})

return normalized

def normalize_azure(azure_resources):

normalized = []

for res in azure_resources:

normalized.append({

"resource_name": res["name"],

"resource_type": res["type"],

"location": res["location"]

})

return normalized

Step 5: Validation Logic (Drift Detection)

def compare_resources(excel_data, tf_data, azure_data):

issues = []

for excel_res in excel_data:

name = excel_res['resource_name']

tf_match = next((r for r in tf_data if r['resource_name'] == name), None)

az_match = next((r for r in azure_data if r['resource_name'] == name), None)

if not tf_match:

issues.append({

"resource": name,

"issue": "Missing in Terraform",

"severity": "High"

})

if not az_match:

issues.append({

"resource": name,

"issue": "Missing in Azure",

"severity": "Critical"

})

if tf_match and az_match:

if excel_res['region'] != az_match.get('location'):

issues.append({

"resource": name,

"issue": "Region mismatch",

"expected": excel_res['region'],

"actual": az_match.get('location')

})

return issues

drift_report = compare_resources(

normalize_excel(excel_df),

normalize_tf(tf_resources),

normalize_azure(azure_resources)

)

print(drift_report)

Step 6: Export Report to Excel

Sample validation

Resource	Issue	Expected	Actual	Severity
func-app-01	Missing in Terraform	-	-	High
search-01	SKU mismatch	Standard	Basic	Medium
webapp-01	Region mismatch	East US	West Europe	High

Building Multi-File Refactoring Agents with GitHub Copilot Workspace

Devi_Priya — Tue, 05 May 2026 03:49:40 GMT

Introduction

AI-assisted development continues to evolve beyond inline code suggestions toward end-to-end engineering workflows. While tools such as GitHub Copilot have significantly improved developer productivity at the function and file level, modern applications demand capabilities that operate across the entire repository.

Refactoring, modernization, and architectural changes rarely occur in isolation. They require coordinated updates across multiple files, services, and layers.

With GitHub Copilot Workspace, developers can now move from incremental edits to intent-driven, multi-file transformations powered by AI.

This article walks through:

The role of Copilot Workspace in modern development workflows
How to access and use the Workspace experience
A practical, end-to-end refactoring scenario
Key benefits and considerations for enterprise adoption

From Code Assistance to Code Orchestration

Traditional AI-assisted development focuses on generating code snippets in response to local context. While effective, this approach is limited when tasks require:

Cross-file consistency
Architectural alignment
Large-scale refactoring

Copilot Workspace introduces a different model:

Developers define intent, and AI orchestrates repository-wide execution.

This enables a shift toward:

Task-oriented development
Structured planning before execution
Coordinated multi-file updates

Getting Started with Copilot Workspace

Access to GitHub Copilot Workspace depends on feature availability and organizational enablement. The following entry points are commonly used:

Access from a Repository

Navigate to a repository in GitHub
Select the Copilot option or Open in Workspace

Direct Workspace Access

You can also navigate to:

https://github.com/copilot/workspace

If enabled, this opens the Workspace interface.

Understanding the Workspace Experience

Copilot Workspace provides a structured interface designed for task execution:

Intent Panel – Define the desired outcome
Planning View – Review AI-generated steps
Multi-file Editor – Inspect and refine changes
Execution Controls – Apply updates and create pull requests

This workflow emphasizes transparency and control, ensuring developers remain in the loop.

Key Benefits

Repository-Aware Intelligence

Copilot Workspace analyzes relationships across files, enabling more accurate and consistent transformations.

Intent-Driven Workflows

Developers focus on what needs to be done, while the system determines how to execute it.

Consistent Multi-File Updates

Changes are applied uniformly across controllers, services, and supporting components.

Accelerated Refactoring and Modernization

Large-scale changes can be executed efficiently, reducing manual effort and risk.

Practical Scenario: Modernizing Authentication

To illustrate the capabilities of Copilot Workspace, consider a common enterprise scenario:

An application currently uses password-based authentication, implemented across multiple layers. The goal is to migrate to a token-based authentication model using a centralized service.

Initial State

Authentication logic is distributed across the application:

ValidateUser(username, password)

This pattern appears in:

Controllers
Services
Middleware

Defining the Intent

Within GitHub Copilot Workspace, the developer provides a structured instruction:

Replace all password-based authentication with token-based authentication using AuthService. Update all references, introduce dependency injection, and ensure consistency across the application.

AI-Generated Plan

Copilot Workspace analyzes the repository and produces a plan that includes:

Identifying all usages of ValidateUser
Introducing a centralized authentication service
Updating controllers to return tokens
Refactoring middleware for token validation
Configuring dependency injection

This plan provides a transparent view of the proposed transformation.

Refactored State

Centralized Authentication Service

public interface IAuthService { string GenerateToken(string username); bool ValidateToken(string token); }

public class AuthService : IAuthService { public string GenerateToken(string username) { return Convert.ToBase64String(Encoding.UTF8.GetBytes(username)); } public bool ValidateToken(string token) { return !string.IsNullOrEmpty(token); } }

Updated Controller

public class UserController : Controller { private readonly IAuthService _authService; public UserController(IAuthService authService) { _authService = authService; } public IActionResult Login(string username, string password) { var token = _authService.GenerateToken(username); return Ok(new { Token = token }); }

Updated Middleware

public class AuthMiddleware { private readonly RequestDelegate _next; private readonly IAuthService _authService; public AuthMiddleware(RequestDelegate next, IAuthService authService) { _next = next; _authService = authService; } public async Task Invoke(HttpContext context) { var token = context.Request.Headers["Authorization"]; if (!_authService.ValidateToken(token)) { context.Response.StatusCode = 401; return; } await _next(context); } }

Dependency Injection

services.AddScoped<IAuthService, AuthService>();

Outcome

The transformation delivers measurable improvements:

Centralized authentication logic
Improved security model
Consistent implementation across layers
Reduced technical debt

Best Practices for Adoption

To maximize effectiveness:

Provide clear, structured intent
Review generated plans before execution
Validate changes through testing and code review
Start with targeted scenarios before scaling

Conclusion

GitHub Copilot Workspace represents a meaningful advancement in AI-assisted development. By enabling developers to define intent and delegate execution, it supports repository-wide transformations with greater consistency and efficiency.

As development workflows continue to evolve, tools that combine context awareness, planning, and execution will play a central role in modern engineering practices.

Build and Deploy Logic App Workflows Using Visual Studio Code and CI/CD Pipeline

Devi_Priya — Tue, 05 May 2026 03:39:26 GMT

Throughout this guide, you'll create a Standard logic app workspace and project, build your workflow, and deploy it as a Standard logic app resource in Azure. This enables your workflow to run in a single-tenant Azure Logic Apps environment or within an App Service Environment v3 (restricted to Windows-based App Service plans).

Key advantages of Standard logic apps include:

You can locally develop, debug, run, and test workflows within the Visual Studio Code environment. Although both the Azure portal and Visual Studio Code support building, running, and deploying Standard logic app resources and workflows, Visual Studio Code allows you to perform all these actions locally, offering greater flexibility during development.

Prerequisites

Visual Studio Code
Azure Account extension for Visual Studio Code
Download and install the following Visual Studio Code dependencies for your specific operating system using either method:

Install all dependencies manually --> For manual installation
Install all dependencies automatically.

Starting with version 2.81.5, the Azure Logic Apps (Standard) extension for Visual Studio Code includes a dependency installer that automatically installs all the required dependencies in a new binary folder and leaves any existing dependencies unchanged.

For more information, see Get started more easily with the Azure Logic Apps (Standard) extension for Visual Studio Code.

This extension includes the following dependencies:

Dependency	Description
C# for Visual Studio Code	Enables F5 functionality to run your workflow.
Azurite for Visual Studio Code	Provides a local data store and emulator to use with Visual Studio Code so that you can work on your logic app project and run your workflows in your local development environment. If you don't want Azurite to automatically start, you can disable this option: 1. On the File menu, select Preferences > Settings. 2. On the User tab, select Extensions > Azure Logic Apps (Standard). 3. Find the setting named Azure Logic Apps Standard: Auto Start Azurite, and clear the selected checkbox.
.NET SDK 6.x.x	Includes the .NET Runtime 6.x.x, a prerequisite for the Azure Logic Apps (Standard) runtime.
Azure Functions Core Tools - 4.x version	Installs the version based on your operating system (Windows, macOS, or Linux). These tools include a version of the same runtime that powers the Azure Functions runtime, which the Azure Logic Apps (Standard) extension uses in Visual Studio Code.
Node.js version 16.x.x unless a newer version is already installed	Required to enable the Inline Code Operations action that runs JavaScript.

Set up Visual Studio code

To make sure that all the extensions are correctly installed, reload or restart Visual Studio Code.

Confirm that the Azure Logic Apps Standard: Project Runtime setting for the Azure Logic Apps (Standard) extension is set to version ~4:
On the File menu, go to Preferences > Settings.
On the User tab, go to > Extensions > Azure Logic Apps (Standard).
You can find the Azure Logic Apps Standard: Project Runtime setting here or use the search box to find other settings:

Connect to your Azure account

On the Visual Studio Code Activity Bar, select the Azure icon.

In the Azure window, on the Workspace section toolbar, from the Azure Logic Apps menu, select Create New Project.

From the templates list that appears, select either Stateful Workflowor Stateless Workflow.

Provide a name for your workflow and press Enter.

If Visual Studio Code prompts you to open your project in the current Visual Studio Code or in a new Visual Studio Code window, select Open in current window.

Visual Studio Code finishes creating your project.

The Explorer pane shows your project, which now includes automatically generated project files. For example, the project has a folder that shows your workflow's name. Inside this folder, the workflow.json file contains your workflow's underlying JSON definition.

Open the workflow.json file's shortcut menu, and select Open Designer.

If it asks for Enable connectors in Azure, select Use connectors from Azure

After the Select subscription list opens, select the Azure subscription to use for your logic app project.
After the resource groups list opens, select RG to use for your logic app project.
After you perform this step, Visual Studio Code opens the workflow designer.

After you open a blank workflow in the designer, the Add a trigger prompt appears on the designer. You can now start creating your workflow by adding a trigger and actions and save it.

Run, test, and debug locally

Make sure to start the emulator before you run your workflow:
In Visual Studio Code, from the View menu, select Command Palette.
After the command palette appears, enter Azurite: Start.
On the Visual Studio Code Activity Bar, open the Run menu, and select Start Debugging (F5).

The Terminal window opens so that you can review the debugging session.

Now, find the callback URL for the endpoint on the Request trigger.

Reopen the Explorer pane so that you can view your project.
From the jsonfile's shortcut menu, select Overview.

Click on Run trigger

If it is stateful workflow, you’ll be able to see the status as shown below.

To view it, click on identifier.

It will open a new window with the results.

Note: Incase while using storage account in your workflow if you get any forbidden error then whitelist your IP in that storage account and rerun the workflow by choosing Run and debug in VS code.

Upon completion stop the debug by choosing the stop button and push the code to azure repo using git commands to push the code.

Use a Pipeline to Deploy the Created Workflow

Build.yaml

jobs: - job: logic_app_build displayName: "Build and publish Logic App" steps: - script: sudo apt-get update && sudo apt-get install -y zip displayName: 'Install zip utility' - task: CopyFiles@2 displayName: 'Create project folder' inputs: sourceFolder: '$(System.DefaultWorkingDirectory)' contents: | azure_logicapps/** targetFolder: 'project_output' - task: ArchiveFiles@2 displayName: 'Create project Zip' inputs: rootFolderOrFile: '$(System.DefaultWorkingDirectory)/project_output/azure_logicapps' includeRootFolder: false archiveType: 'zip' archiveFile: '$(Build.ArtifactStagingDirectory)/$(Build.BuildId).zip' replaceExistingArchive: true - task: PublishPipelineArtifact@1 displayName: 'Publish project zip artifact' inputs: targetPath: '$(Build.ArtifactStagingDirectory)/$(Build.BuildId).zip' artifact: 'logicAppCIArtifact' publishLocation: 'pipeline'

Deploy.yaml

jobs: - deployment: deploy_logicapp_resources displayName: Deploy Logic App environment: ${{ parameters.environmentToDeploy }} strategy: runOnce: deploy: steps: - download: current artifact: logicAppCIArtifact - task: AzureFunctionApp@1 displayName: 'Deploy Logic App workflows' inputs: azureSubscription: ${{ parameters.azureServiceConnection }} appType: 'functionApp' appName: ${{ parameters.vars.LogicAppName }} package: '$(Pipeline.Workspace)/logicAppCIArtifact/$(Build.BuildId).zip' deploymentMethod: 'zipDeploy'

Running GitHub Actions Runners on Azure Container Apps with KEDA Autoscaling

shubhijain — Mon, 04 May 2026 13:03:30 GMT

GitHub-hosted runners work well for most scenarios. But as workloads grow, teams often need:

Better cost optimization — pay only when jobs run
More control over execution environments and installed tools
Scalable parallel execution — run 10, 20, or 50 jobs simultaneously
Network access to private resources (databases, internal APIs)

Traditionally, this is solved using self-hosted runners on Virtual Machines. But VMs come with challenges — always-on cost, manual scaling, patching, and maintenance overhead.

In this guide, we'll walk through a modern, serverless alternative:

👉 Running self-hosted GitHub runners on Azure Container Apps Jobs with KEDA autoscaling

What You Will Build

By the end of this guide:

Capability	What You Get
Runner type	Self-hosted, ephemeral (one job = one container)
Scaling	Automatic via KEDA — scales to zero when idle
Cost	Zero cost when no jobs are running
Infrastructure	Fully managed by Azure Container Apps
Security	Secrets stored in Azure Key Vault, Managed Identity for auth

Architecture Overview

Here's how the system works at a high level:

How It Works (Step by Step):

A developer pushes code or triggers a GitHub Actions workflow
GitHub queues the job and looks for a runner with matching labels
KEDA (built into Container Apps) polls the GitHub Actions API for pending jobs
When a pending job is detected, KEDA triggers the Container App Job to start a new execution
A fresh container starts, registers itself as a self-hosted runner with GitHub
The runner picks up the job, executes it, and reports results back to GitHub
The container shuts down and is destroyed — fully ephemeral
When no jobs are pending, KEDA scales back to zero — no cost

Runtime Flow

Pre-requisites

Before you begin, make sure you have:

Requirement	Details
GitHub account	With a repository or organization where you want to run workflows
Azure subscription	With permissions to create resources (Contributor role or higher)
Azure CLI	Installed locally, OR use Azure Cloud Shell (no install needed)
Basic knowledge	Familiarity with GitHub Actions and Azure Portal

Where You'll Run Commands

Throughout this guide, you'll need a terminal to create files and run CLI commands. You have three options:

Option	When to Use	How to Open
VS Code Terminal (Recommended)	You have VS Code installed locally	Open VS Code → Ctrl + `` (backtick) → Terminal opens at the bottom
Azure Cloud Shell	No local tools installed, or restricted machine	Go to portal.azure.com → click the >_ icon in the top toolbar
Any terminal	PowerShell, CMD, Bash — whatever you prefer	Just ensure Azure CLI (az) is installed

💡recommended to use VS Code because you'll create files (Dockerfile, start.sh) AND run commands — VS Code lets you do both in one place.

Azure Resources We Will Create

Resource	Purpose
Resource Group	Logical container for all resources
Azure Container Registry (ACR)	Stores the runner Docker image
Azure Container Apps Environment	Hosting environment for container jobs
Azure Container App Job	The actual runner job definition
Azure Key Vault	Securely stores the GitHub PAT token
Managed Identity	Allows the container job to access ACR and Key Vault without passwords

💡 Note: This guide covers organization-level runners. For repository-level runners, the only difference is the GitHub API endpoint used for registration. We'll call out the differences where applicable.

Step 1: Create a GitHub Personal Access Token (PAT)

Before we touch Azure, we need a token that allows our runner to register with GitHub. You can use either a Fine-grained token (recommended) or a Classic token.

Option A: Fine-Grained PAT (Recommended)

Fine-grained tokens let you scope access to specific repositories only. This is critical for two reasons:

Avoid GitHub API rate limits: KEDA continuously polls the GitHub API for pending jobs. If your token has access to your entire org (potentially hundreds of repos), KEDA scans all of them on every polling cycle. GitHub allows only 5,000 API requests/hour — with broad access, you'll hit this limit quickly and KEDA will stop detecting jobs.
Security: Least-privilege access — the token only works on the repos you explicitly select.

🔑 This is why we recommend fine-grained tokens over classic tokens. By selecting only the repos that need runners, KEDA polls fewer repos and stays well within API limits.

Go to github.com → Settings → Developer settings
Click Personal access tokens → Fine-grained tokens
Click Generate new token
Fill in:

Field	Value
Token name	container-app-runner
Expiration	Choose based on your needs (e.g., 90 days)
Resource owner	Your GitHub username or org
Repository access	Only select repositories → pick the repos where you want runners

⚠️ IMPORTANT — Remember these repo names! The repos you select here are the ONLY repos this token can access. Later in Step 8, when you configure the KEDA scale rule, you must list these exact same repos in the repos metadata field. KEDA uses this token to poll GitHub for pending jobs — if a repo isn't included in the token, KEDA can't see its jobs and your runners won't scale for it.

Example: If you select my-app and my-api here, your KEDA config must have repos: my-app,my-api.

Under Permissions, set the following:

Repository permissions (required):

Permission	Access Level	Why It's Needed
Actions	Read and write	Manage workflow runs and artifacts
Administration	Read and write	Register and manage self-hosted runners
Metadata	Read-only	(Auto-selected, required)
Workflows	Read and write	Update GitHub Action workflow files

Organization permissions (only if using org-level runners):

Permission	Access Level	Why It's Needed
Self-hosted runners	Read and write	Register runners at the org level

📝 For personal accounts (no org): You only need the Repository permissions above. Skip the Organization permissions.

Click Generate token
⚠️ IMPORTANT: Copy the token NOW — you won't be able to see it again!

Option B: Classic PAT

If you prefer a classic token (simpler but broader access):

Go to Settings → Developer settings → Personal access tokens → Tokens (classic)
Click Generate new token (classic)
Select these scopes:

Scope	Why It's Needed
✅ repo	Full access to repositories
✅ workflow	Manage workflows
✅ admin:org	Required for org-level runners (skip for personal repos)

Click Generate token and copy it immediately

⚠️ Classic tokens give access to ALL repositories in your account. For better security and to avoid API rate limits, prefer fine-grained tokens scoped to specific repos.

Save the token somewhere safe temporarily. We'll store it in Azure Key Vault in the next steps.

Where Will Runners Appear on GitHub?

Once runners are deployed and register with GitHub, you can see them here:

For repository-level runners:

Go to your repo → Settings → Actions → Runners

For organization-level runners:

Go to your org → Settings → Actions → Runners

Runners automatically register themselves when the container starts. You'll see them appear as "Idle" or "Active" in the Runners list. You do NOT need to manually create individual runners on GitHub.

Setting Up Runner Groups (Organization-Level) — Recommended

Since this guide is for organization-level runners, it's recommended to create a Runner Group on GitHub. Runner Groups let you control which repositories in your org can use these runners.

Steps to Create a Runner Group:

Go to your GitHub Organization page (e.g., https://github.com/your-org)
Click Settings (top menu bar)
In the left sidebar, expand Actions → click Runner groups
Click New runner group
Fill in:

Field	Value
Name	container-app-runners (or any descriptive name)
Repository access	Choose one:
	• All repositories — any repo in the org can use these runners
	• Selected repositories — pick specific repos (recommended for control)
Allow public repositories	Uncheck this for security (unless needed)
Workflow access	Leave default (all workflows)

⚠️ CRITICAL: If ANY of the repositories using these runners are PUBLIC, you MUST check "Allow public repositories". If this is unchecked, GitHub will silently refuse to dispatch jobs from public repos to runners in this group — the runner will register and show as "Idle", but workflows will stay stuck in "Queued" forever. This is the most common and hardest-to-debug issue with runner groups.

Click Create group

Why Create a Runner Group?

Without Runner Group	With Runner Group
Any repo in the org can use your runners	Only selected repos can use them
Harder to track which teams use runners	Clear visibility and access control
Potential security risk for public repos	Can block public repos from using runners

📝 For personal GitHub accounts: Runner groups are not available. Runners will automatically appear under your repo's Settings → Actions → Runners. No extra setup needed.

Step 2: Create Azure Resources (Portal)

We'll create all the Azure infrastructure through the Azure Portal. If you prefer CLI, see the CLI alternative at the end.

⚠️ IMPORTANT: All Azure resources (Resource Group, ACR, Key Vault, Container Apps Environment, Container App Job) must be in the SAME region. Pick one region (e.g., Central US or West US 2) and use it for everything. Mixing regions can cause connectivity issues and increased latency.

2.1: Create a Resource Group

Go to Azure Portal
Search for "Resource groups" in the top search bar
Click + Create
Fill in:

Field	Value
Subscription	Select your Azure subscription
Resource group	rg-github-runners (or your preferred name)
Region	West US 2 (or any region that supports Container Apps)

Click Review + create → Create

2.2: Create an Azure Container Registry (ACR)

This is where we'll store our runner Docker image.

Search for "Container registries" in the Azure Portal
Click + Create
Fill in:

Field	Value
Subscription	Your subscription
Resource group	rg-github-runners
Registry name	yourregistryname (must be globally unique, lowercase, letters and numbers only)
Location	Same as your resource group (e.g., West US 2)
Pricing plan	Basic (sufficient for this guide; use Standard/Premium for production)

Click Review + create → Create
Once created, note down the Login server (e.g., yourregistryname.azurecr.io) — you'll need this later

2.3: Create an Azure Key Vault

Search for "Key vaults" in the Azure Portal
Click + Create
Fill in:

Field	Value
Subscription	Your subscription
Resource group	rg-github-runners
Key vault name	kv-github-runners (must be globally unique)
Region	Same region
Pricing tier	Standard

Go to the Access configuration tab:
- Select Azure role-based access control (RBAC) as the permission model
Click Review + create → Create

2.4: Store the GitHub PAT in Key Vault

Open your newly created Key Vault
First, give yourself permission:
- Go to Access Control (IAM) → + Add role assignment
- Role: Key Vault Secrets Officer
- Assign to: Your own Azure account
- Click Review + assign
Now go to Objects → Secrets → + Generate/Import
Fill in:

Field	Value
Upload options	Manual
Name	github-pat
Secret value	Paste your GitHub PAT from Step 1

Click Create

Step 3: Create the Runner Docker Image

This is the Docker image that will run as your GitHub Actions runner. We need two files: a Dockerfile and a start.sh script.

3.1: Set Up Your Working Directory

Open VS Code on your local machine
Open the integrated terminal: Ctrl + ` (backtick) or Terminal → New Terminal
Create a new folder and navigate into it:

mkdir github-runner-image cd github-runner-image

You'll create two files in this folder: Dockerfile and start.sh
In VS Code, click File → Open Folder and open the github-runner-image folder (so you can edit files easily)

📌 All commands in Step 3 and Step 4 should be run from inside this github-runner-image folder.

3.2: Choose Your Approach

There are two approaches to creating the runner image:

Approach	Best For	Docker Required Locally?
Option A: Build locally with Docker	Development/testing	✅ Yes
Option B: Build remotely with ACR Tasks	Production / no Docker access	❌ No

We'll create the same files for both approaches. The only difference is the build command.

3.3: Create the start.sh Script

This script runs when the container starts. It registers the runner with GitHub, executes the job, and then the container shuts down.

Create a file named start.sh:

#!/bin/bash set -e # ──────────────────────────────────────────── # CONFIGURATION # ──────────────────────────────────────────── # These values are passed as environment variables # GITHUB_PAT → Your GitHub Personal Access Token # GITHUB_OWNER → Your GitHub org or username # GITHUB_REPO → (Optional) Repository name for repo-level runners # RUNNER_SCOPE → "org" or "repo" # RUNNER_LABELS → Comma-separated labels (e.g., "container-app,linux") # RUNNER_GROUP → Runner group name (org-level only, default: "Default") # Set this to the runner group you created in Step 1 (e.g., "container-app-runners") # If not set, runners register in GitHub's "Default" group — which means # ANY repo in the org can use them and you lose access control. RUNNER_SCOPE="${RUNNER_SCOPE:-org}" RUNNER_LABELS="${RUNNER_LABELS:-container-app}" RUNNER_GROUP="${RUNNER_GROUP:-Default}" # ⚠️ IMPORTANT: Always set the RUNNER_GROUP environment variable on your Container App Job # to match the runner group you created on GitHub (e.g., "container-app-runners"). # The "Default" fallback above is only a safety net — do NOT rely on it. # ──────────────────────────────────────────── # GET REGISTRATION TOKEN # ──────────────────────────────────────────── if [ "$RUNNER_SCOPE" == "org" ]; then echo "🔑 Requesting registration token for organization: $GITHUB_OWNER" REG_TOKEN=$(curl -s -X POST \ -H "Authorization: token $GITHUB_PAT" \ -H "Accept: application/vnd.github+json" \ "https://api.github.com/orgs/${GITHUB_OWNER}/actions/runners/registration-token" \ | jq -r .token) RUNNER_URL="https://github.com/${GITHUB_OWNER}" else echo "🔑 Requesting registration token for repository: $GITHUB_OWNER/$GITHUB_REPO" REG_TOKEN=$(curl -s -X POST \ -H "Authorization: token $GITHUB_PAT" \ -H "Accept: application/vnd.github+json" \ "https://api.github.com/repos/${GITHUB_OWNER}/${GITHUB_REPO}/actions/runners/registration-token" \ | jq -r .token) RUNNER_URL="https://github.com/${GITHUB_OWNER}/${GITHUB_REPO}" fi if [ -z "$REG_TOKEN" ] || [ "$REG_TOKEN" == "null" ]; then echo "❌ Failed to get registration token. Check your GITHUB_PAT and permissions." exit 1 fi echo "✅ Registration token obtained successfully" # ──────────────────────────────────────────── # CONFIGURE RUNNER # ──────────────────────────────────────────── echo "⚙️ Configuring runner..." ./config.sh --unattended \ --name "runner-$(hostname)" \ --url "$RUNNER_URL" \ --token "$REG_TOKEN" \ --runnergroup "$RUNNER_GROUP" \ --ephemeral \ --labels "$RUNNER_LABELS" \ --replace echo "🚀 Starting runner..." ./run.sh

Key flags explained:

--ephemeral: Runner processes one job then exits (container stops)
--runnergroup: Registers the runner in a specific runner group (org-level only)
--replace: Replaces any existing runner with the same name
--unattended: No interactive prompts

⚠️ Do NOT use --disableupdate! In newer GitHub versions, this flag prevents GitHub from dispatching jobs to the runner. The runner will appear as "Idle" but never pick up work.

3.4: Create the Dockerfile

Create a file named Dockerfile:

FROM ubuntu:22.04 # Prevent interactive prompts during package installation ENV DEBIAN_FRONTEND=noninteractive # Install required dependencies RUN apt-get update && apt-get install -y \ curl \ git \ jq \ ca-certificates \ unzip \ wget \ apt-transport-https \ software-properties-common \ && rm -rf /var/lib/apt/lists/* # Create a non-root user for the runner (GitHub requires this) RUN useradd -m runner # Set up the runner directory WORKDIR /home/runner/actions-runner # Download the latest GitHub Actions Runner # Check latest version: curl -s https://api.github.com/repos/actions/runner/releases/latest | jq -r '.tag_name' ARG RUNNER_VERSION=2.334.0 RUN curl -L -o actions-runner.tar.gz \ "https://github.com/actions/runner/releases/download/v${RUNNER_VERSION}/actions-runner-linux-x64-${RUNNER_VERSION}.tar.gz" \ && tar xzf actions-runner.tar.gz \ && rm actions-runner.tar.gz # Install runner dependencies RUN ./bin/installdependencies.sh # Copy the startup script COPY start.sh . RUN chmod +x start.sh # Set ownership to the runner user RUN chown -R runner:runner /home/runner # Switch to non-root user USER runner ENTRYPOINT ["./start.sh"]

💡 Tip: Check the latest runner version by running:

curl -s https://api.github.com/repos/actions/runner/releases/latest | jq -r '.tag_name'

At the time of writing, 2.334.0 is the latest. Update the ARG RUNNER_VERSION value if a newer version is available.

⚠️ Using a deprecated runner version will cause runners to connect but refuse to pick up jobs. You'll see the error: "Runner version vX.X.X is deprecated and cannot receive messages." Always use a recent version.

Step 4: Build and Push the Docker Image

Now we need to build this image and push it to your Azure Container Registry (ACR).

Option A: Build Locally with Docker (Development)

Use this if you have Docker Desktop installed on your machine.

Where to run: In the VS Code terminal (or any terminal), make sure you're inside the github-runner-image folder where your Dockerfile and start.sh are located.

# 0. First, log in to Azure (this opens a browser window for authentication) az login # 1. Log in to your ACR (replace yourregistryname with your actual ACR name) az acr login --name yourregistryname # 2. Build the image docker build -t yourregistryname.azurecr.io/github-runner:v1 . # 3. Push the image to ACR docker push yourregistryname.azurecr.io/github-runner:v1

📌 Make sure Docker Desktop is running before you execute these commands. If you see "Cannot connect to the Docker daemon", start Docker Desktop first.

🔒 Don't have az login or Docker on your machine? Use Azure Cloud Shell instead — it's a browser-based terminal at shell.azure.com that comes pre-authenticated with Azure CLI (no az login needed) and has Docker available. See Option B below if you can't use Docker at all.

Option B: Build Remotely with ACR Tasks (Production — No Docker Needed) ⭐

This is the recommended approach for production environments where:

You don't have Docker installed
Your production environment is fully private / locked down
You want to build images directly in Azure without any local tooling

Where to run:

VS Code terminal → Run az login first, then the build command
Azure Cloud Shell (shell.azure.com) → No az login needed, you're already authenticated. Upload your Dockerfile and start.sh files using the Upload button in Cloud Shell, then run the build command

You must be inside the folder where your Dockerfile and start.sh are located.

# If running from VS Code terminal (skip this line if using Azure Cloud Shell) az login # Build directly in ACR — no Docker required! az acr build --registry yourregistryname --image github-runner:v1 --file Dockerfile .

☝️ That's it — one single command. No Docker install, no Docker daemon, nothing.

Using Azure Cloud Shell? To upload files:

Open shell.azure.com
Click the Upload/Download button (📁 icon) in the toolbar
Upload Dockerfile and start.sh
They'll land in your home directory (~/). Run az acr build from there.

This command:

Uploads your source code to ACR
Builds the Docker image in Azure (not on your machine)
Tags and stores it in your registry
No Docker daemon needed at all!

🔒 Production Note: In locked-down environments where even az acr build isn't possible from your machine, you can:

Use Azure Cloud Shell (browser-based, always has Azure CLI)
Set up an ACR Task with a Git trigger — ACR automatically builds when you push to a repo
Use Azure DevOps / GitHub Actions pipeline to build and push the image

Example: Auto-build from a GitHub repo (single line for Cloud Shell)

az acr task create --registry yourregistryname --name build-runner-image --image "github-runner:{{.Run.ID}}" --context https://github.com/your-org/your-runner-repo.git --file Dockerfile --git-access-token YOUR_GITHUB_PAT

Verify the Image

After building, confirm the image exists in your registry:

Portal: Go to your ACR → Repositories → you should see github-runner listed

CLI:

az acr repository list --name yourregistryname --output table

Step 5: Create the Container Apps Environment

The Container Apps Environment is the hosting platform for your container jobs. Think of it as the "cluster" where your runners will live.

Steps (Azure Portal):

Search for "Container Apps Environment" in the Azure Portal
Click + Create
Fill in:

Tab	Field	Value
Basics	Subscription	Your subscription
	Resource group	rg-github-runners
	Environment name	cae-github-runners
	Region	West US 2 (same as other resources)
	Environment type	Consumption only (or Consumption + Dedicated if you need workload profiles)
Monitoring	Log Analytics workspace	Create new or select existing

Leave Networking as defaults for now (we'll discuss production networking later)
Click Review + create → Create

⏳ This takes 1-2 minutes to create.

Step 6: Create the Container App Job

This is the core resource — the Container App Job that will run your GitHub runners.

⚠️ IMPORTANT: Complete Step 4 (Build & Push Image) BEFORE this step. The Container App Job creation form requires you to select a container image from your ACR. If your ACR is empty (no images pushed), the portal will show an error: "The ACR does not have any images. Please push an image to the ACR and try again." So make sure your image is pushed first!

Steps (Azure Portal):

Search for "Container App Jobs" in the Azure Portal search bar
Click + Create Container App Job

Tab 1: Basics

Field	Value	Notes
Subscription	Your subscription
Resource group	rg-github-runners	Same RG as other resources
Container app job name	github-runner-job	Lowercase, letters, numbers, hyphens
Region	West US 2	Must match your CAE region
Container Apps Environment	cae-github-runners	Select the environment created in Step 5

Click Next: Container >

Tab 2: Container

Field	Value	Notes
Name	github-runner	Name of the container within the job
Image source	Azure Container Registry	Select this radio button
Registry	yourregistryname.azurecr.io	Select your ACR from the dropdown
Image	github-runner	Select the image you pushed
Image tag	v1	The tag you used during build
Registry authentication	Managed identity	Leave as default
Managed identity	System assigned Identity (environment)	Leave as default — Azure will auto-assign ACR Pull role
Command override	(Leave empty)	Our Dockerfile already has an ENTRYPOINT
Arguments override	(Leave empty)	Not needed
Workload profile	Consumption	Default is fine
CPU and memory	0.5 CPU cores, 1 Gi memory	Increase if your jobs need more resources

💡 CPU/Memory guidance:

0.5 CPU / 1 Gi — Light jobs (linting, simple tests)
1 CPU / 2 Gi — Medium jobs (building apps, running test suites)
2 CPU / 4 Gi — Heavy jobs (compiling large projects, ML workloads)

Environment Variables

Click + Add for each variable:

Name	Source	Value
GITHUB_OWNER	Manual	Your GitHub org name (e.g., Quality-Framework)
RUNNER_SCOPE	Manual	org (or repo for repo-level runners)
RUNNER_LABELS	Manual	container-app
GITHUB_REPO	Manual	Comma-separated repo names that this runner will serve (e.g., my-app,my-api,my-infra). These should match the repos selected in your PAT (Step 1).
RUNNER_GROUP	Manual	The GitHub runner group name (e.g., container-app-runners). Must match the group created in Step 1. If not set, defaults to Default.
GITHUB_PAT	(We'll configure this as a secret reference in Step 7)

⚠️ Do NOT put the PAT directly as an environment variable value. We will securely reference it from Key Vault in Step 7.

For now, add only GITHUB_OWNER, RUNNER_SCOPE, RUNNER_LABELS, GITHUB_REPO, and RUNNER_GROUP. We'll add GITHUB_PAT after setting up the identity and secret.

⚠️ Make sure RUNNER_SCOPE matches runnerScope in the scale rule below. If you're using org-level runners, both should be org.

Scale Rule Settings (same page, scroll down)

Below the environment variables, you'll see Scale rule settings:

Field	Value	Notes
Min executions	0	Scale to zero when no jobs are pending
Max executions	5	Maximum parallel runners (adjust as needed). Do NOT set to 0 — this means unlimited and can cause hundreds of runner containers to spawn!
Polling interval	30	How often (in seconds) KEDA checks for pending jobs

Scale Rules — Click + Add

A side panel opens with the "Add scale rule" form. Fill in:

Field	Value
Rule name	github-runner-rule
Custom rule type	github-runner

Scale parameters (key-value pairs — click + Add for each):

Name	Value	Notes
githubAPIURL	https://api.github.com	Pre-filled by the portal, leave as-is
owner	your-github-org	Your GitHub org or username (e.g., Quality-Framework)
runnerScope	org	org for org-level, repo for repo-level
repos	your-repo-name	Must match the repos you selected in your PAT (Step 1). Comma-separated, no spaces.
targetWorkflowQueueLength	1	Number of pending jobs needed to trigger one runner
labels	container-app	Must match RUNNER_LABELS env var and runs-on in your workflow

⚠️ CRITICAL: Do NOT skip the labels parameter! Without it, KEDA cannot match pending jobs to your runner and will always show MetricValue: 0.00 — meaning no containers will ever start. This is the most common setup mistake.

Authentication:

Secret reference	Trigger parameter
(Leave empty for now — we'll configure this after creating the job and setting up Key Vault secrets in Step 7)	personalAccessToken

💡 The portal may let you save the scale rule without a secret reference. If it blocks you, delete the Authentication row entirely, click Add scale rule, and we'll add the authentication after the job is created.

Click Add scale rule → Then click Review + create → Create

⏳ Creation takes about 1 minute.

⚠️ First-time creation may fail with an image pull error if you're creating a new Container Apps Environment at the same time. This happens because the environment's managed identity gets the AcrPull role during deployment, but the image pull happens before the role propagates. If this occurs, simply Redeploy or create the Container App Job again — the role is already assigned and the second attempt will succeed.

Step 7: Configure Managed Identity and Secrets

Now we need to:

Enable Managed Identity on the Container App Job
Grant it access to Key Vault and ACR
Reference the GitHub PAT as a secret

7.1: Enable System-Assigned Managed Identity

Open your Container App Job (github-runner-job)
In the left menu, go to Settings → Identity
Under System assigned, toggle Status to On
Click Save → Click Yes to confirm
Note the Object ID that appears — you'll need this

7.2: Grant Key Vault Access

Go to your Key Vault (kv-github-runners)
Go to Access Control (IAM) → + Add role assignment
Fill in:

Field	Value
Role	Key Vault Secrets User
Assign access to	Managed identity
Members	Search for github-runner-job and select it

Click Review + assign

7.3: Grant ACR Pull Access

Go to your Container Registry (yourregistryname)
Go to Access Control (IAM) → + Add role assignment
Fill in:

Field	Value
Role	AcrPull
Assign access to	Managed identity
Members	Search for github-runner-job and select it

Click Review + assign

7.4: Add GitHub PAT as a Secret Reference

Go back to your Container App Job (github-runner-job)
In the left menu, go to Settings → Secrets
Click + Add
Fill in:

Field	Value
Type	Key Vault reference
Key	github-pat
Key Vault secret URL	Select your Key Vault and the github-pat secret
Managed Identity	System assigned

Click Add

7.5: Map the Secret to an Environment Variable

Go to Settings → Containers
Click on your container → Edit
Go to the Environment variables tab
Click + Add:

Name	Source	Value
GITHUB_PAT	Reference a secret	github-pat

Click Save

💡 If the Save button is greyed out, try making a small edit to another field first (e.g., click into a value and click out) to trigger the save state. Alternatively, re-create the container with the correct env vars.

Step 8: Configure KEDA Scale Rule (GitHub Runner Scaler)

This is where the magic happens. KEDA's GitHub Runner scaler monitors the GitHub Actions API for pending workflow jobs and scales your Container App Job accordingly.

Steps:

Open your Container App Job (github-runner-job)
Go to Settings → Scale (or Scale and replicas)
Under Scale rule, click + Add
Fill in the scale rule:

Field	Value	Notes
Name	github-runner-rule	Any descriptive name
Type	Custom	Select Custom
Custom rule type	github-runner	This is the KEDA scaler type

Under Metadata, add these key-value pairs:

Key	Value	Notes
owner	your-github-org	Your GitHub org or username (e.g., Quality-Framework)
repos	repo1,repo2	Must match the repos you selected in your PAT token (Step 1). Comma-separated, no spaces. E.g., qualityframework-demo,qualityframework-bicep. Do NOT leave empty — if left empty, KEDA scans ALL org repos and you'll hit GitHub API rate limits.
runnerScope	org	org for org-level runners, repo for repo-level
labels	container-app	Must match the RUNNER_LABELS env var and the runs-on labels in your workflow YAML
targetWorkflowQueueLength	1	Number of pending jobs needed to trigger one new runner instance

Under Authentication, add:

Key	Value
Secret reference	github-pat
Trigger parameter	personalAccessToken

Click Add → Save

Understanding the Scale Rule

Scenario	KEDA Action
0 pending jobs with container-app label	0 runners (scale to zero)
1 pending job	Starts 1 container
3 pending jobs	Starts 3 containers (up to max)
Jobs complete	Containers stop, scale back to zero

Step 9: Test the Setup

9.1: Create a Test Workflow

In any repository within your GitHub organization, create a workflow file:

File: .github/workflows/test-container-runner.yml

name: Test Container App Runner on: workflow_dispatch: # Allows manual trigger from GitHub UI push: branches: [main] jobs: test-runner: runs-on: [self-hosted, container-app] # Must match your RUNNER_LABELS steps: - name: Checkout code uses: actions/checkout@v4 - name: Print runner info run: | echo "✅ Hello from Azure Container App Runner!" echo "Runner Name: $RUNNER_NAME" echo "Runner OS: $RUNNER_OS" echo "Workspace: $GITHUB_WORKSPACE" - name: Run a simple test run: | echo "Current directory: $(pwd)" echo "Files in repo:" ls -la echo "System info:" uname -a echo "Memory:" free -h

9.2: Trigger the Workflow

Go to your repository on GitHub
Click Actions tab
Select "Test Container App Runner" from the left sidebar
Click Run workflow → Run workflow

9.3: Watch It Work

In the GitHub Actions tab, you'll see the job show as "Queued"
In Azure Portal, go to your Container App Job → Execution history
Within ~30 seconds (your polling interval), you should see a new execution start
The job will:
- Container starts → Runner registers → Job executes → Container stops
Back in GitHub, the workflow run should show as ✅ completed

Troubleshooting

Problem	Likely Cause	Fix
Job stays "Queued" forever	KEDA not detecting jobs	Check scale rule metadata — ensure labels, owner, repos, runnerScope are all filled correctly
Container starts but workflow doesn't complete	Wrong secret reference or empty env vars	Verify GITHUB_PAT env var points to correct secret, and all env vars have values
"Permission denied" errors	PAT missing required scopes	Edit PAT and add missing permissions (Step 1)
Image pull errors on first deploy	ACR access timing issue	Redeploy — the AcrPull role was assigned but hadn't propagated yet
Runner registers but job doesn't run	Label mismatch	Ensure runs-on labels in workflow match RUNNER_LABELS env var AND labels in KEDA scale rule
Runner shows "Idle" but job stays queued	--disableupdate flag or wrong runner group	Remove --disableupdate from start.sh and rebuild the image. Also verify RUNNER_GROUP env var matches the runner group name on GitHub, and the group has the correct repos assigned
Runner version deprecated error	Outdated runner binary	Update RUNNER_VERSION in Dockerfile to the latest version and rebuild. Run curl -s https://api.github.com/repos/actions/runner/releases/latest \| jq -r '.tag_name' to check
Hundreds of offline runners spawning	maxExecutions set to 0 (unlimited)	Set maxExecutions to a reasonable limit (e.g., 5 or 10) in the scale rule settings
Runner idle + job queued forever (public repo)	Runner group blocks public repos	Go to Org Settings → Actions → Runner groups → your group and check "Allow public repositories". Without this, GitHub silently refuses to dispatch jobs from public repos to runners in the group

To check container logs:

Go to your Container App Job → Monitoring → Logs and run:

ContainerAppConsoleLogs_CL | where ContainerGroupName_s startswith "github-runner-job" | where TimeGenerated > ago(30m) | order by TimeGenerated desc | take 20

Production Considerations

🔒 Networking: Private Environments

In production, your Container Apps Environment may be deployed inside a VNet with no public internet access. Here's how to handle that:

Private ACR Access

Container App Job ──(private endpoint)──► ACR

Create a Private Endpoint for your ACR
Disable public access on ACR
Ensure your Container Apps Environment is in the same (or peered) VNet

Private Key Vault Access

Same pattern — create a Private Endpoint for Key Vault and disable public access.

GitHub API Access

Your runner containers need outbound access to:

github.com (runner registration)
api.github.com (KEDA polling)
*.actions.githubusercontent.com (downloading actions)

If using a firewall or NSG, ensure these are allowed.

🔐 Using GitHub App Instead of PAT (Recommended for Production)

PATs are tied to individual users and have broad scopes. For production, consider using a GitHub App:

Create a GitHub App in your organization
Grant it Organization Self-hosted runners: Read & Write permissions
Install the app in your organization
Use the App ID and Private Key instead of PAT

The KEDA GitHub Runner scaler supports GitHub App authentication natively.

📊 Monitoring and Alerting

Set up monitoring for your runners:

Container App Job Metrics (Azure Monitor):
- Execution count
- Execution duration
- Failed executions
Alerts to set up:
- Alert when executions fail repeatedly
- Alert when execution queue is growing (KEDA can't keep up)
- Alert when runner registration fails (PAT expired?)
Log Analytics queries:

// Find failed executions ContainerAppConsoleLogs_CL | where ContainerGroupName_s startswith "github-runner-job" | where Log_s contains "error" or Log_s contains "failed" | order by TimeGenerated desc

💰 Cost Optimization

Strategy	Impact
Scale to zero (min: 0)	No cost when idle
Right-size CPU/memory	Don't over-provision
Set reasonable max executions	Prevent runaway costs
Use Consumption plan	Pay per-second billing
Set replica timeout	Kill stuck jobs

🔄 Keeping the Runner Image Updated

Runner versions get outdated. Set up automated rebuilds:

# Create a scheduled ACR Task to rebuild weekly (single line for Cloud Shell) az acr task create --registry yourregistryname --name rebuild-runner-weekly --image github-runner:latest --context https://github.com/your-org/runner-image-repo.git --file Dockerfile --schedule "0 0 * * 0" --git-access-token YOUR_PAT

Appendix A: CLI Commands for All Steps

If you prefer CLI over Portal, here are all the commands. Run these in Azure Cloud Shell or any terminal with Azure CLI.

All commands are written as single lines so they work directly in Azure Cloud Shell without formatting issues.

# ─── Set your variables (update these values) ─── RESOURCE_GROUP="rg-github-runners" LOCATION="westus2" ACR_NAME="yourregistryname" KV_NAME="kvgithubrunners" CAE_NAME="cae-github-runners" JOB_NAME="github-runner-job" IMAGE_NAME="github-runner" IMAGE_TAG="v1" GITHUB_ORG="your-github-org" # ─── Step 1: Create Resource Group ─── az group create --name $RESOURCE_GROUP --location $LOCATION # ─── Step 2: Create ACR ─── az acr create --name $ACR_NAME --resource-group $RESOURCE_GROUP --sku Basic # ─── Step 3: Create Key Vault ─── az keyvault create --name $KV_NAME --resource-group $RESOURCE_GROUP --location $LOCATION --enable-rbac-authorization # ─── Step 4: Store PAT in Key Vault ─── az keyvault secret set --vault-name $KV_NAME --name "github-pat" --value "YOUR_PAT_HERE" # ─── Step 5: Build Image with ACR Tasks (run from the folder with Dockerfile) ─── az acr build --registry $ACR_NAME --image $IMAGE_NAME:$IMAGE_TAG . # ─── Step 6: Create Container Apps Environment ─── az containerapp env create --name $CAE_NAME --resource-group $RESOURCE_GROUP --location $LOCATION # ─── Step 7: Create Container App Job (single command) ─── az containerapp job create --name $JOB_NAME --resource-group $RESOURCE_GROUP --environment $CAE_NAME --trigger-type Event --replica-timeout 1800 --replica-retry-limit 1 --replica-completion-count 1 --parallelism 1 --image "$ACR_NAME.azurecr.io/$IMAGE_NAME:$IMAGE_TAG" --cpu "0.5" --memory "1Gi" --min-executions 0 --max-executions 5 --polling-interval 30 --scale-rule-name "github-runner-rule" --scale-rule-type "github-runner" --scale-rule-metadata "owner=$GITHUB_ORG" "runnerScope=org" "labels=container-app" "targetWorkflowQueueLength=1" --scale-rule-auth "personalAccessToken=github-pat" --secrets "github-pat=keyvaultref:https://$KV_NAME.vault.azure.net/secrets/github-pat,identityref:system" --env-vars "GITHUB_PAT=secretref:github-pat" "GITHUB_OWNER=$GITHUB_ORG" "RUNNER_SCOPE=org" "RUNNER_LABELS=container-app" "RUNNER_GROUP=container-app-runners" --registry-server "$ACR_NAME.azurecr.io" --registry-identity "system"

Appendix B: Repo-Level Runner Changes

If you want runners at the repository level instead of organization level:

Setting	Org-Level	Repo-Level
PAT scope	admin:org	repo only
RUNNER_SCOPE env var	org	repo
GITHUB_REPO env var	Not needed	Required (e.g., my-repo)
KEDA runnerScope	org	repo
KEDA repos	Optional	Required

Summary

Here's what we built:

Component	What It Does
Dockerfile + start.sh	Creates an ephemeral GitHub Actions runner image
ACR	Stores the runner image securely
Key Vault	Stores the GitHub PAT securely
Container Apps Environment	Provides the hosting platform
Container App Job	Runs the runner containers on demand
KEDA Scale Rule	Automatically scales runners based on pending GitHub jobs
Managed Identity	Connects everything securely without passwords

The Result:

✅ Zero cost when idle — no VMs running 24/7 ✅ Automatic scaling — KEDA handles it ✅ Ephemeral runners — clean environment every time ✅ Secure — secrets in Key Vault, Managed Identity for auth ✅ No Docker required — ACR Tasks builds images in the cloud ✅ Production ready — private networking, monitoring, automated image updates

-------------------------------------------------------------------------------------------

Have questions or feedback? Drop a comment below!

Tags: Azure, Container Apps, GitHub Actions, KEDA, Self-hosted Runners, DevOps, Serverless

Modernizing Terraform Pipelines on Azure: OIDC Federation for GitHub Actions and Azure DevOps

ssinghkalra — Sat, 02 May 2026 20:34:19 GMT

The secret nobody wants to rotate

Most Terraform-on-Azure pipelines we see still authenticate the same way they did three years ago. A long-lived ARM_CLIENT_SECRET sitting in GitHub Actions or Azure DevOps, set once, copied around, and rotated only when something breaks.

It's the most ignored credential in the cloud, and statistically the most likely one to leak. A developer screenshots a variable group. A pipeline log echoes a value. A fork inherits a secret. Or the secret simply expires on a Friday evening and takes production deployments with it.

Workload Identity Federation (WIF) makes this whole class of problem go away. The pipeline mints a short-lived token at runtime, exchanges it for an Azure access token via Microsoft Entra, and never touches a secret. GitHub Actions has supported it since 2021. Azure DevOps service connections went GA with WIF in February 2024. The azurerm Terraform provider has supported it since v3.7.

This post walks through the pattern end-to-end, for both GitHub Actions and Azure DevOps, the way I've rolled it out across multiple customer estates.

How the exchange actually works

Before any YAML, it helps to picture what's happening:

The CI system (GitHub or ADO) signs a short-lived JWT describing exactly what's running- which repo, which branch, which environment, which service connection.
The pipeline sends that JWT to Microsoft Entra ID.
Entra checks it against a federated identity credential you've configured on a managed identity or app registration. The iss, sub, and aud claims must match case-sensitively.
If it matches, Entra returns an Azure access token valid for the duration of the job.
Terraform uses it. The job ends. The token expires. Nothing persists.

The token is bound to a specific subject like repo:contoso/platform:environment:prod or sc://contoso/platform/azure-prod. It can't be reused from another repo, branch, or pipeline.

Recommended Architecture

A few choices that usually hold up in production:

Decision	Choice
Identity type	User-assigned managed identity (UAMI), not app registration
Identity granularity	One UAMI per environment (not per pipeline)
Trust scope	Pinned to the environment claim, not the branch
RBAC scope	Resource group, not subscription
Remote state	OIDC + use_azuread_auth = true, shared key access disabled

Why UAMIs? They live in your subscription, don't need Application Administrator rights to manage, and follow the lifecycle of the resource group they belong to. Why one per environment? Pipeline-per-identity explodes into hundreds of identities. Environment-per-identity maps cleanly to deployment scopes.

Part 1 - GitHub Actions

Step 1: Create the identity and federate it

Two commands per environment. That's it.

az identity create -g rg-platform-identity -n id-tf-prod -l eastus az identity federated-credential create \ --name github-prod \ --identity-name id-tf-prod \ --resource-group rg-platform-identity \ --issuer https://token.actions.githubusercontent.com \ --subject repo:contoso/platform:environment:prod \ --audiences api://AzureADTokenExchange

Repeat for nonprod. No secret is created anywhere.

Step 2: Wire it up in GitHub

In repo Settings → Environments, create nonprod and prod. On prod, add required reviewers and a branch rule restricting deployments to main. Then add three environment variables (not secrets - these aren't sensitive): AZURE_CLIENT_ID, AZURE_TENANT_ID, AZURE_SUBSCRIPTION_ID.

The workflow itself stays small:

permissions: id-token: write contents: read jobs: apply: runs-on: ubuntu-latest environment: prod env: ARM_USE_OIDC: "true" ARM_CLIENT_ID: ${{ vars.AZURE_CLIENT_ID }} ARM_TENANT_ID: ${{ vars.AZURE_TENANT_ID }} ARM_SUBSCRIPTION_ID: ${{ vars.AZURE_SUBSCRIPTION_ID }} steps: - uses: actions/checkout@v4 - uses: hashicorp/setup-terraform@v3 - run: terraform init && terraform apply -auto-approve

Three things make this secure:

id-token: write is the only elevated permission, and it doesn't grant write access to anything in GitHub, it just lets the runner mint a JWT.
The environment: line picks the right AZURE_CLIENT_ID and drives the sub claim. The federation refuses anything else.
No azure/login step is needed for Terraform. The azurerm provider reads GitHub's OIDC environment variables automatically.

Part 2 - Azure DevOps

The model is identical. The mechanics are different.

ADO offers two creation paths for a WIF service connection: automatic (it creates an app registration for you) and manual (you bring your own UAMI). For platform teams, manual + UAMI is almost always the better choice to ensure identity lives where governance lives.

The flow is a small dance between the two portals:

In Azure DevOps, create a new ARM service connection → choose Workload Identity Federation (manual) → fill in your UAMI's client ID, tenant ID, and subscription. Save as draft. ADO shows you an issuer URL and a subject identifier.
In Azure, on the UAMI, add a federated credential using the values ADO showed you. The subject looks like sc://contoso/platform/azure-prod.
Back in ADO, click Verify and save.

In the pipeline, the service connection only "activates" if a task in the job loads it. The simplest way is the AzureCLI@2 task:

- task: AzureCLI@2 inputs: azureSubscription: azure-prod # the WIF service connection scriptType: bash scriptLocation: inlineScript inlineScript: | terraform init && terraform apply -auto-approve env: ARM_USE_OIDC: "true" ARM_CLIENT_ID: $(AZURE_CLIENT_ID) ARM_TENANT_ID: $(AZURE_TENANT_ID) ARM_SUBSCRIPTION_ID: $(AZURE_SUBSCRIPTION_ID) ARM_ADO_PIPELINE_SERVICE_CONNECTION_ID: $(SERVICE_CONNECTION_ID) SYSTEM_ACCESSTOKEN: $(System.AccessToken) SYSTEM_OIDCREQUESTURI: $(System.OidcRequestUri)

For teams converting dozens of legacy connections, the Azure DevOps team published a PowerShell helper that walks every ARM service connection in a project and converts them in place. There's a 7-day rollback window on each connection, which makes the migration genuinely low-risk.

Don't forget the state file

The Terraform state is your real blast radius. With OIDC, it's almost free to lock it down too. The same UAMI can read and write blob data without the storage account key:

backend "azurerm" { resource_group_name = "rg-tfstate" storage_account_name = "sttfstateprodeastus" container_name = "platform-prod" key = "platform.tfstate" use_oidc = true use_azuread_auth = true }

Grant the UAMI Storage Blob Data Contributor on the container (not the account), disable shared key access on the storage account, and you've removed the last secret in the pipeline.

RBAC and break-glass

Federation removes a credential, not a privilege. A few habits worth keeping:

Scope role assignments to resource groups, not subscriptions. The whole point of federation is that scoping is now trivially easy.
Use Role Based Access Control Administrator instead of User Access Administrator if your Terraform creates role assignments. It's a more recent, narrower role.
Have a documented break-glass. If GitHub or ADO has a token-service incident, you still need a path to ship a hotfix. A single hardware-key-protected emergency app registration in a separate identity boundary works well, audited monthly.
Monitor sign-ins. Every federated exchange shows up in Entra sign-in logs as a service principal sign-in. Pipe these to Sentinel and alert on anomalies like sign-ins outside expected hours, or from IPs outside GitHub's published ranges.

The errors you will hit (and what they really mean)

Symptom	What it actually is
AADSTS70021: No matching federated identity record found	Case-sensitive mismatch in iss, sub, or aud. Almost always a trailing slash or a capitalised character
AADSTS700016: Application not found in directory	Wrong client ID or tenant. Not a federation problem
403 on a resource even though token exchange worked	Federation is fine. Your RBAC isn't. Check the exact scope
Unable to determine OIDC token (ADO)	No task in the job loaded the service connection. Add an AzureCLI@2 step
Works on main, fails on tags	You pinned sub to a branch ref. Add a second federated credential for tags, or move to environment-based scoping

Migrating without a maintenance window

You almost never get to do this on a greenfield repo. The order that has worked for me on legacy estates:

Create the new UAMI alongside the old service principal, with the same role assignments.
Federate one canary pipeline. Verify it deploys equivalently.
Cut over pipelines in waves, lowest-risk environment first.
Once a full release cycle passes cleanly, disable the old SP's secret.
Wait another cycle. Then delete the SP entirely.
Add a CI gate that fails any new pipeline introducing ARM_CLIENT_SECRET.

The old and new auth methods coexist on the same subscription throughout. There's no hard cutover and no maintenance window, just a steady drift toward zero secrets.

Wrapping up

If you do nothing else after reading this, do one thing: search your CI variable groups for ARM_CLIENT_SECRET. Every result is an outage or a breach waiting to happen.

Federation is one of those rare changes that's both more secure and less work to operate. Once you've set it up, you stop thinking about credential rotation, secret expiry, and quarterly access reviews for service principals. The pipeline simply runs, and the audit trail is in Entra where it belongs.

That's a good trade.

Automating Azure Naming Standards using API and DevOps Pipelines

sameenamohammed — Tue, 05 May 2026 09:43:42 GMT

Introduction

In large Azure environments, one of the most overlooked yet critical governance challenges is resource naming consistency.

While organizations define naming standards, enforcing them at scale across multiple subscriptions, teams, and pipelines often becomes a manual and inconsistent process.

In real-world projects, this leads to:

Operational confusion
Difficult resource identification
Reduced traceability
Governance gaps

To address this, we implemented an API-driven naming validation approach integrated with Azure DevOps pipelines, ensuring every resource created follows organizational standards automatically.

The Problem: Inconsistent Naming Across Environments

In distributed teams and large-scale environments, naming issues commonly arise due to:

Multiple developers creating resources independently
Lack of centralized enforcement
Manual validation during deployments
No integration with CI/CD pipeline

Example (Before Automation)

Resource Type	Example Name
Resource Group	testRG1
Storage Account	mystorage123
VM	vm-prod

Problems:

No standard structure
No environment or region context
Hard to manage at scale

Goal

To ensure:

✅ Standardized naming across all resources
✅ Automated validation during deployments
✅ No manual intervention required
✅ Seamless integration with DevOps workflows

Solution Overview

We implemented a naming enforcement mechanism using:

Azure Naming Tool (or similar API-based naming service)
Azure DevOps Pipelines
Managed Identity for secure authentication

Architecture Flow

Automated Naming Validation using Naming API, Managed Identity, and DevOps Pipeline

🔍 Solution Flow Explained

Developer Commit
The process begins when a developer commits code to the repository, triggering the Azure DevOps pipeline.
Azure DevOps Pipeline Execution
The pipeline runs deployment scripts as part of the CI/CD process.
Managed Identity Authentication
The pipeline uses Managed Identity to securely authenticate and obtain an access token—eliminating the need for storing credentials.
Naming API Invocation
A request is sent to the Naming API with resource details such as:
- Resource type
- Environment
- Location
- Application name
Validation & Name Generation
The Naming API validates inputs and returns a compliant resource name based on predefined standards.
Deployment Decision
- If validation succeeds → resources are deployed
- If validation fails → deployment is blocked
Resource Deployment
Only validated, compliant resources are provisioned in Azure.

Note:
The “Azure Naming API” referenced in this blog represents an implementation pattern rather than a native Azure service.
Solutions such as Azure Naming Tool or custom APIs can be used to expose naming logic and integrate with DevOps pipelines for automated enforcement. This approach can be implemented using solutions like Azure Naming Tool, Resource Name Generator, or custom-built APIs

Implementation Details

Authentication using Managed Identity

To securely access the Naming API:

Managed Identity is used
No secrets or credentials stored in pipeline
Token retrieved dynamically

PowerShell Implementation

Below is a simplified version of what was used in implementation:

# Get access token using Managed Identity $token = (Get-AzAccessToken -ResourceUrl "api://NamingTool").Token # Call naming API $response = Invoke-RestMethod ` -Uri "https://your-namingtool-api-endpoint/api/naming" ` -Headers @{ Authorization = "Bearer $token" } ` -Method POST ` -Body @{ resourceType = "resourceGroup" environment = "prod" location = "eastus" application = "app01" } | ConvertTo-Json # Extract generated resource name $resourceName = $response.name Write-Output "Generated Name: $resourceName"

🔄 Azure DevOps Pipeline Integration

Naming validation is integrated directly into the deployment pipeline. Sample Pipeline Snippet

- task: AzureCLI@2 inputs: azureSubscription: 'ServiceConnection' scriptType: 'ps' scriptLocation: 'inlineScript' inlineScript: | Write-Output "Calling Naming API" .\scripts\Get-ResourceName.ps1

Key Benefit:

👉 Resource names are validated before deployment, preventing non-compliant resources from being created.

Security Considerations

Use Managed Identity for API authentication
Avoid storing secrets in pipelines
Ensure API access is restricted and secured

Extending This Solution

This approach can be extended to:

Enforcing tagging standards
Policy validation before deployment
Subscription vending automation
Cost governance controls

✨ Final Thoughts

Naming standards are often documented—but rarely enforced effectively.

By integrating API-based naming validation into DevOps pipelines, organizations can move from:

Guidelines → ✅ Automated Enforcement

This ensures governance is:

Scalable
Consistent
Developer-friendly

Reimagining Azure Governance with Automation & EPAC

sameenamohammed — Sat, 02 May 2026 19:26:11 GMT

🧩 The Challenge: Governance at Scale

Managing Azure environments manually introduces:

❌ Policy drift across subscriptions
❌ Inconsistent naming conventions
❌ Delays in compliance enforcement
❌ Human errors in deployments

🔍 Insight: Governance gaps are often not due to lack of policies—but lack of automation.

Solution Overview: EPAC

Enterprise Policy as Code (EPAC) to bring governance into the DevOps workflow.

EPAC helped us:

Treat policies like code
Automate deployments
Standardize governance
Maintain audit history

Architecture Overview

Our EPAC setup included:

Management Groups hierarchy for governance scope
Policy Definitions & Initiatives (JSON/Bicep)
Azure DevOps Pipeline for deployment
Managed Identity for secure execution

This is sample implementation flow of EPAC in enterprises

Flow Explanation

Azure DevOps Pipeline
- Triggers CI/CD process
- Executes deployment scripts
- Authenticates securely using Managed Identity
EPAC Framework
- Stores policies as code
- Enables version control and validation
- Acts as the central governance engine
Azure Policy Engine
- Evaluates resources against defined policies
- Enforces compliance automatically
Target Environments
- Policies applied across:
  - Management Groups
  - Subscriptions
  - Resource Groups

Why Policy-as-Code Matters

✅ Standardized governance across environments
✅ Faster onboarding of subscriptions
✅ Improved audit readiness
✅ Repeatable and reliable policy deployment
✅ Seamless DevOps integration

Security & Compliance Benefits

Enforces least privilege access
Prevents misconfigured deployments
Supports continuous compliance
Aligns with standards like ISO 27001

Implementation Walkthrough

✅ Step 1: Structuring Policy Repository

This structure ensured:

Clear separation of concerns
Reusability
Easy onboarding for new contributors

✅ Step 2: Defining Policies

We created custom policies and reused built-in ones.

Example: Restrict allowed regions

{

"if": {

"field": "location",

"notIn": ["eastus", "westeurope"]

"then": {

"effect": "deny"

}

✅ Step 3: Creating Initiatives

Instead of assigning individual policies, we grouped them into initiatives:

Security baseline
Tagging compliance
Cost optimization

👉 This reduced duplication and simplified assignments.

✅ Step 4: Pipeline Automation

We built an Azure DevOps pipeline to:

Validate policy templates
Deploy definitions to management groups
Assign initiatives automatically

Example pipeline flow:

This is example pipeline structure for deploying epac policies targeted to single environment

parameters: - name: forceDeployment displayName: 'Force deployment (ignore change detection)' type: boolean default: false - name: clearAgentCache displayName: 'Clear agent container cache (recommended for troubleshooting)' type: boolean default: true variables: # Pipeline per il deployment delle Policy in ambiente Canary/Test PAC_OUTPUT_FOLDER: ./Output PAC_DEFINITIONS_FOLDER: ./Definitions # Service connection per l'ambiente di test/canary serviceConnection: "SC-EPAC-CONTRIBUTOR-TST-001" # Environment selector per canary pacEnvironmentSelector: canary # Trigger: deploy solo manualmente o da branch specifici trigger: none # PR trigger per validazione pr: branches: include: - main - feature/* paths: include: - src/IaC/Infrastructure/epac/Definitions/* pool: name: "TST-AgentPool-01" stages: - stage: Plan displayName: "Plan Canary Environment" jobs: - job: Plan displayName: "Generate Deployment Plan" steps: - template: templates/plan.yml parameters: serviceConnection: $(serviceConnection) pacEnvironmentSelector: ${{ variables.pacEnvironmentSelector }} - stage: Deploy displayName: "Deploy to Canary (Audit Mode)" dependsOn: Plan condition: and(not(failed()), not(canceled()), or(eq('${{ parameters.forceDeployment }}', 'true'), and(eq('${{ parameters.forceDeployment }}', 'false'), or(eq(dependencies.Plan.outputs['Plan.Plan.deployPolicyChanges'], 'yes'), eq(dependencies.Plan.outputs['Plan.Plan.deployRoleChanges'], 'yes'))))) variables: PAC_INPUT_FOLDER: "$(Pipeline.Workspace)/plans-${{ variables.pacEnvironmentSelector }}" localDeployPolicyChanges: $[stageDependencies.Plan.Plan.outputs['Plan.deployPolicyChanges']] localDeployRoleChanges: $[stageDependencies.Plan.Plan.outputs['Plan.deployRoleChanges']] jobs: - deployment: DeployPolicy displayName: "Deploy Policy Changes (Audit Mode)" environment: PAC-CANARY condition: and(not(failed()), not(canceled()), or(eq('${{ parameters.forceDeployment }}', 'true'), and(eq('${{ parameters.forceDeployment }}', 'false'), eq(variables.localDeployPolicyChanges, 'yes')))) strategy: runOnce: deploy: steps: - template: templates/deploy-policy.yml parameters: serviceConnection: $(serviceConnection) pacEnvironmentSelector: ${{ variables.pacEnvironmentSelector }} forceDeployment: ${{ parameters.forceDeployment }} - deployment: DeployRoles displayName: "Deploy Role Assignments" dependsOn: DeployPolicy environment: PAC-CANARY condition: and(not(failed()), not(canceled()), eq(variables.localDeployRoleChanges, 'yes')) # Riabilitato per AMBA managed identity strategy: runOnce: deploy: steps: - template: templates/deploy-roles.yml parameters: serviceConnection: $(serviceConnection) pacEnvironmentSelector: ${{ variables.pacEnvironmentSelector }} # Stage opzionale per validazione post-deployment - stage: Validate displayName: "Validate Canary Deployment" dependsOn: Deploy condition: and(succeeded(), ne(variables['Build.Reason'], 'PullRequest')) jobs: - job: ValidateCompliance displayName: "Validate Policy Compliance" steps: - task: PowerShell@2 displayName: "Check Policy Compliance Status" inputs: targetType: 'inline' script: | Write-Host "##[section]Canary deployment completed successfully" Write-Host "##[warning]Remember: All policies are in AUDIT mode - monitor compliance dashboard" Write-Host "##[task.complete result=Succeeded;]Canary validation completed"

✅ Step 5: Secure Deployment using Managed Identity

We used:

System-assigned Managed Identity
RBAC roles (Policy Contributor / Reader)

✅ Benefit:

No secrets in pipeline
Improved security posture

✅ Step 6: Policy Assignment at Scale

Policies were assigned at:

Root Management Group
Subscription level (when needed)

This ensured:

Consistent enforcement
Centralized control

Real Use Cases Implemented

Using EPAC, we solved real scenarios:

🔹 Enforcing naming conventions
🔹 Ensuring mandatory resource tagging
🔹 Restricting deployment regions
🔹 Enforcing backup policies on disks
🔹 Preventing creation of non-compliant resources

Best Practices

Start with baseline policies first
Use initiatives instead of individual assignments
Enable PR-based approvals
Always test policies in lower environments
Maintain clear documentation

Conclusion

Implementing EPAC transformed our governance model from manual and reactive → automated and proactive.

For teams managing complex Azure environments, EPAC provides:

Scalability
Consistency
Security

If you are still managing policies manually, this is the right time to: 👉 Move to Policy as Code

AI‑Accelerated AVM Refactoring: Modernizing Legacy IaC Safely and Swiftly

HimanshuYadav — Fri, 01 May 2026 12:14:29 GMT

Why AVM refactoring is harder in brownfield

Greenfield AVM adoption is straightforward: pick the module, deploy, and iterate. Brownfield is different. You’re refactoring *live* infrastructure — where state, naming conventions, diagnostic settings, and policy baselines already exist.

In one anonymized engagement, an AI‑assisted audit found that **43% of module invocations (~10 modules) ** still relied on legacy, non‑AVM wrappers — even though modernization work was already underway.

That number wasn’t just a KPI. It became the roadmap: which modules to prioritize, where inconsistency risk was highest, and which components needed deeper plan review.

The hidden risks you only feel during refactor

AVM adoption often introduces more than a module source change. It can also bring:

New naming patterns (especially around extensions like diagnostics)
More structured configuration objects (maps/objects replacing flat inputs)
Additional ‘helper’ resources (RBAC, wait timers, diagnostics wiring)
Policy‑aligned defaults (encryption, public access disabled, logging enabled)
Provider/version constraint pressure (to meet AVM expectations)

None of these are bad — in fact, they’re usually improvements. But in a live estate, each change must be checked through one lens: **state safety**.

Where AI actually helps (and where it doesn’t)

AI is most valuable when it reduces *mechanical effort* — the repetitive work that slows teams down — while humans keep ownership of architecture and risk.

1) Automated codebase audit (the fastest win)

AI can scan a repo and produce an inventory of module usage, versions, and adoption status: direct AVM, AVM‑wrapped, legacy wrappers, and native resources. This turns hours of manual inspection into a structured baseline report.

2) Draft refactors scaffolding (with mandatory human review)

Tools like GitHub Copilot can generate first‑pass Terraform refactors — reshaping legacy calls into AVM‑style interfaces and scaffolding optional blocks (diagnostics, identities, RBAC).

But AI output should be treated as *a proposal*, not a truth. The most dangerous failures aren’t syntax errors — they’re subtle mismatches: a parameter mapped to the wrong field, a default that changes behavior, or an omitted lifecycle constraint.

3) Terraform plan diff interpretation (explain what changed)

Refactoring to AVM can expand plan output. AI can help summarize large diffs into:

Benign additions (telemetry wiring, diagnostics scaffolding, RBAC helpers)
Behavioral changes requiring sign‑off (network exposure, encryption posture, identity model)
High‑risk actions (destroy/recreate) and the exact resource addresses involved

This doesn’t replace plan review — it accelerates understanding so reviewers can focus on what truly matters.

4) Policy violation translation (from red to ready)

When policy gates fail (Azure Policy, Checkov, etc.), AI is great at translating requirements into actionable remediation — and checking whether AVM supports it natively or needs supplementary configuration.

5) Repository hygiene enforcement (structure as a hard gate)

In multi‑team repos, drift happens: ad‑hoc scripts, local module copies, inconsistent folder patterns. AI can continuously scan for these anti‑patterns and flag deviations early — before they become ‘how we do it now’.

6) Specification‑driven development (the future‑proof approach)

Microsoft’s AVM guidance now explicitly discusses **AI‑assisted IaC solution development** — pairing AVM modules with AI tools to speed delivery while keeping humans in control. In parallel, approaches like Spec Kit promote structured, specification‑driven workflows so requirements and constraints remain the source of truth.

The operating model that keeps you safe

Here’s the simplest rule I’ve seen work consistently:

**AI drafts. Humans validate. The Terraform plan decides.**

That operating model prevents two extremes: (1) refusing AI because it’s imperfect, and (2) trusting AI output blindly because it sounds confident.

A practical AI‑accelerated refactor playbook

If you want a repeatable approach that scales across environments, here’s a playbook that balances speed and safety:

Baseline audit: inventory module sources and adoption categories.
Equivalence check: identify AVM‑ready modules vs AVM gaps.
Slice the work: refactor one bounded component first.
Use AI for scaffolding: generate draft code and a migration checklist.
Plan review discipline: categorize additions vs updates vs replacements.
Import decision framework: import where state safety matters; accept in‑place updates only when semantics are unchanged.
Governance gates: enforce structure + policy + plan review before merge.
Iterate: expect multiple cycles — AI should compress cycles, not eliminate them.

Don’t couple refactoring with compliance

One more lesson worth calling out: **AVM adoption and compliance are related, but not identical.**

Treat policy enforcement as a continuous pipeline requirement from Dev through Prod — independent of whether a component is fully AVM‑aligned. This avoids scope creep (“refactor means fix everything”) while still driving the estate toward a no‑surprises posture.

What ‘success’ looks like

A successful AI‑accelerated AVM refactor typically delivers:

Lower variance between environments
Fewer one‑off wrappers and exceptions
Stronger defaults aligned to policy
A smaller drift surface area
Faster, safer change velocity

And the best part? It changes the mindset from ‘IaC as scripts’ to **IaC as a governed product** — with standards that hold up in audits and operations.

Closing thought

AI won’t replace your architectural accountability — and it shouldn’t try to. But it *can* remove the friction that makes refactoring feel impossible.

If you’re sitting on a pile of legacy wrapper modules today, consider this: the safest time to modernize is **before** the next urgent change lands in your backlog.

References (public)

Azure Verified Modules (AVM): https://azure.github.io/Azure-Verified-Modules/
AI‑Assisted IaC Solution Development (AVM): https://azure.github.io/Azure-Verified-Modules/experimental/ai-assisted-sol-dev/
Spec Kit (AVM): https://azure.github.io/Azure-Verified-Modules/experimental/ai-assisted-sol-dev/spec-kit/
AVM Telemetry guidance: https://azure.github.io/Azure-Verified-Modules/help-support/telemetry/
Microsoft Learn – Azure Verified Modules overview: https://learn.microsoft.com/en-us/community/content/azure-verified-modules

🚀 From Drift to Diagnosis: AI‑Powered Root Cause Analysis for Azure Infrastructure

Pooja_Pradhan — Thu, 30 Apr 2026 07:08:53 GMT

🌍 A Real-World Scenario

During a recent production deployment of an enterprise AI platform, everything looked perfectly aligned from an infrastructure perspective:

✅ Infrastructure deployed via IaC (Terraform)
✅ Private endpoints enforced
✅ Public access disabled for all AI services

A few hours later, an alert triggered.

❗ The Azure OpenAI endpoint was publicly accessible.

This was unexpected — and risky.

🔍 What the team did next

Ran terraform plan → ✅ Drift detected
Checked Azure Portal → ✅ Configuration mismatch confirmed
Reviewed activity logs → ❓ Multiple changes found, but unclear ownership

🚫 The problem

Drift detection tools clearly showed:

“Configuration mismatch”

But they did NOT answer:

Why was it changed?
Who made the change?
Was this intentional or accidental?
What is the impact?
What should be done next?

👉 It took hours of manual investigation to produce a root cause analysis.

💡 The Shift: From Detection to Diagnosis

Most tools today stop at detection.

But what teams really need is:

✅ A system that explains why drift happened and what to do next

This is where AI-powered drift analysis becomes powerful.

🏗️ Architecture Overview

Below is a simple architecture that combines Azure data sources with AI to generate human-readable RCA reports.

🔧 How It Works (Step-by-Step)

✅ Step 1 — Detect Drift

Using standard IaC tools:

Terraform

terraform plan

Bicep

az deployment group what-if </span>

--resource-group rg-ai </span>

--template-file main.bicep

✅ Step 2 — Capture Actual State

Query Azure using Resource Graph:

Resources

| project id, name, type, location, properties

✅ Step 3 — Add Context (Critical Step)

Drift without context is incomplete.

Use Activity Logs:

AzureActivity

| where TimeGenerated > ago(24h)

| project TimeGenerated, ResourceId, OperationName, Caller

👉 This gives:

Who made the change
What operation was executed
When it happened

✅ Step 4 — AI-Powered RCA

Instead of analyzing raw JSON manually, pass the structured data to an AI model.

📥 Input to AI

{

"resource": "openai-endpoint-prod",

"expected": {

"publicNetworkAccess": "Disabled"

"actual": {

"publicNetworkAccess": "Enabled"

"activityLog": {

"caller": "admin@company.com",

"operation": "write",

"time": "2026-04-28T10:15:00Z"

}

🤖 AI Output (Human-Readable RCA)

Drift Summary:

The OpenAI endpoint has public access enabled, which deviates from the expected secure configuration.

Root Cause:

A manual configuration change was performed by admin@company.com via Azure Portal.

Impact:

- Increased exposure to public internet

- Potential violation of security baseline

Recommended Action:

- Revert configuration using IaC deployment

- Apply Azure Policy to enforce private access

- Restrict access using RBAC/PIM

👉 This replaces manual debugging with instant diagnosis.

📊 Drift Digest (Operational View)

Instead of reacting to issues, teams can generate a periodic report:

Resource	Drift Type	Risk	Root Cause	Action
OpenAI Endpoint	Network Exposure	🔴 High	Portal change	Revert + Policy
Storage Account	Security Drift	🔴 High	Script update	Validate automation
Key Vault	RBAC Drift	🔴 Critical	Manual access	Audit roles

⚡ Real-World Drift Scenarios

From enterprise Azure AI implementations:

Private endpoints removed for debugging
Public access enabled temporarily
RBAC permissions added for testing
NSG rules changed for connectivity

👉 These changes are common — and easy to miss.

✅ Best Practices

Always combine:
- IaC state
- Resource Graph
- Activity Logs
Avoid auto-remediation without validation
Use:
- Azure Policy (prevent drift)
- RBAC + PIM (limit access)
- Resource locks (protect critical resources)
Generate a weekly drift digest instead of reactive troubleshooting

💡 Key Takeaway

Drift detection tells you what changed

✅ AI tells you why it changed and what to do

🚀 Looking Ahead

This approach opens new possibilities:

AI-generated incident reports
Drift-aware Copilot assistants
Preventive controls before deployment

🔥 Next in Series

👉 AI Change Risk Scoring for Infrastructure Deployments — Predicting failures before they happen

✍️ Final Thoughts

In modern Azure environments, drift is inevitable.

But with the right combination of:

Observability
Context
Intelligence

👉 Drift becomes not a problem, but a source of insight.

From Drift to Self-Healing: Building a Multi-Repo Azure AI Infrastructure You Can Actually Trust

Valini_Sunthwal — Mon, 04 May 2026 09:58:14 GMT

A question before we begin

Quick — what version of infrastructure is running in your production subscription right now?

Not what your pipeline last deployed. Not what your tracking spreadsheet says. What's actually running — right now, this second — and can you prove it?

If you had to think about it, you're in good company. So did we. That's basically why we ended up building everything I'm about to describe.

The Friday night that started all this

So here's what happened. Production had a broken NSG rule. One of our engineers woke up, logged into the Azure Portal, edited the rule, hit Save. Done. Went back to bed.

Fair enough — production was healthy again.

The problem? Nobody noticed until Monday that someone had changed live infrastructure directly in the portal. No pipeline involved. No Terraform. No tracking. Our deployment registry still showed the old config. For three days, our "source of truth" was just... wrong.

And that's the thing people don't talk about enough with cloud infrastructure. It's not just Terraform drift. It's anyone making a change through the Portal, CLI, PowerShell, REST API — basically any path that isn't your pipeline. Our system catches all of it now, but I'll get to that.

Why this is harder than it sounds

Let me paint the picture. A single Azure environment has easily 20+ resources. VNets with subnets and NSGs. Key Vaults. Storage Accounts with private endpoints. Container App Environments. AI services — OpenAI, AI Foundry. DNS zones, route tables, firewalls, managed identities, RBAC bindings tying everything together.

Each resource has dozens of properties. And each property can be changed through the Portal, CLI, PowerShell, ARM, Bicep, Terraform, or raw REST calls. Any of those changes happen without your pipeline having a clue.

Now multiply that across lab, non-live, and live. Across multiple subscriptions. Across multiple markets. Yeah.

What we mean by "market"

We use the word market to mean a country or regional business unit. Each one runs in its own Azure region with its own subscription. UK deploys to UK South, Czech to Germany West Central, and so on. Same platform blueprint, different geography, different network space, different secrets, different deployment lifecycle.

Think of it as completely isolated copies of the same thing.

The actual problem we had

We run Azure AI infrastructure across multiple markets. Each market has three environments — lab, non-live, live. Each environment has 20+ resources. So we're talking hundreds of resources across dozens of subscriptions, all deployed through Terraform, all expected to stay in sync.

They don't. Pipelines fail halfway through. Someone fixes something manually in the portal because it's 2 AM and they just want it to work. A registry update fails silently. And slowly, what you think is deployed and what's actually deployed start to diverge. One quiet failure at a time.

We needed something different. Not just a deployment pipeline — we needed a system that knows what it deployed, can check whether that's still true, and notices when someone changes something outside the pipeline.

Can it actually detect portal changes?

Yes. But there's a nuance.

When someone edits a resource through the Azure Portal — modifies an NSG rule, changes a Key Vault policy, scales a Container App — the actual Azure resource changes. But Terraform's state file doesn't update because Terraform wasn't involved. The state serial stays the same. The version tracker stays the same. The registry stays the same.

So the daily reconciliation pipeline, which compares serials and versions, would not catch this. All three tracking files still agree — they're just all wrong.

The portal change gets caught the next time the pipeline runs terraform plan. Terraform talks to Azure, compares what's in state versus what's actually there, and shows the diff:

~ resource "azurerm_network_security_rule" "example" {
    ~ access = "Allow" -> "Deny"   # changed outside of Terraform
  }

Then terraform apply reverts the change — brings Azure back in line with the code. That's the self-healing bit.

So in short:

The reconciliation pipeline catches pipeline-level drift — someone ran terraform outside the pipeline, or a registry write failed.
The next terraform plan/apply catches resource-level drift — portal changes, CLI changes, anything outside Terraform.

Between the two, nothing stays hidden for long.

Three repos, one platform

We split things into three repositories. Each does one thing well.

1. Platform Repo

Deploys the foundation — VNets, DNS, Key Vaults, Container App Environments, AI Foundry, managed identities, firewall rules. Everything a market needs before any application team can deploy anything. Triggered by ServiceNow tickets or code merges.

2. Modules Repo

37 reusable Terraform modules, all built on top of Azure Verified Modules. Both the Platform Repo and Use-Case Repo pull from this. The whole point is that if the same Key Vault definition lives in two places, it will diverge within three months. This makes that impossible.

Everything is version-pinned. No silent updates:

source = "git::https://...//avm_modules/keyvault?ref=avm-res-keyvault-vault/v0.10.2"

3. Use-Case Repo

This is where application teams deploy their stuff — storage, databases, function apps, AI Search, container apps — on top of what the Platform Repo already set up. The repo has pre-written Terraform for 20+ resources. Teams don't write Terraform. They uncomment what they need. Push to a feature branch, it deploys to lab. Merge to staging, it goes to non-live. Merge to main, it goes to live. Approval gates at each step.

The five pipelines

Release pipeline. Fires on merge to main or from a ServiceNow trigger. Runs semantic-release, creates a version tag (like v7.0.0), then deploys to lab → non-live → live with an approval gate before live.

Feature pipeline. Lab-only sandbox. Test your changes without touching anything real. Creates soft tags like lab-v7.0.0.1 so you can track what was deployed without cluttering the main version history.

Rollback pipeline. Pick any previous tag (say v6.1.0) and it restores it. The version tracker marks this as a deliberate rollback so reconciliation doesn't flag it as drift.

Reconciliation pipeline. Runs at 6 AM every day. Compares what the registry says is deployed against what's actually in the Terraform state. Catches pipeline-level drift — someone ran terraform apply outside the pipeline, or a registry write failed. Portal changes get caught on the next plan/apply run instead.

Commitlint pipeline. Enforces conventional commit messages. That's what makes semantic versioning work — fix: bumps patch, feat: bumps minor, BREAKING CHANGE: bumps major.

The three tracking files

This is where most platforms stop too early. We keep three independent records for every deployment.

1. Terraform State File

Terraform's own record of what it manages. Stored in Azure Blob Storage. Has a serial number that increments every time Terraform runs. If someone runs terraform apply outside the pipeline, this serial changes but nothing else does.

2. Version Tracker

A JSON file sitting next to the state file. Written by the pipeline after every successful deploy. Records the version tag, commit SHA, run ID, timestamp. If someone deploys outside the pipeline, this file doesn't update — that's how we spot the mismatch.

3. Subscription Registry

A central blob container where each market gets a subscription.json. Records what the platform thinks is deployed in each environment. But if the blob write fails (network blip), it can be wrong. That's why we cross-check against the other two.

When all three agree, we're good. When they don't, we know exactly what drifted, where, and why.

How drift detection works in practice

Every morning at 6, the reconciliation pipeline checks every market, subscription, and environment.

It uses three tiers:

Tier 1 (best case): Version tracker exists. Compare its serial against the actual state serial and the registry. If anything's off, flag it.

Tier 2 (no tracker): Older environments without a version tracker. Fall back to comparing the registry's recorded serial against the actual state serial.

Tier 3 (no serials at all): Compare timestamps. If the state was modified more than five minutes after the last registry update, flag it.

When it finds drift, it does not auto-fix. It generates a report:

Customer     | Environment | Registry  | Tracker   | State Serial | Status
customer-cz  | non-live    | v6.1.0    | v7.0.0    | 45           | DRIFT
customer-cz  | live        | v7.0.0    | v7.0.0    | 41           | OK
customer-uk  | lab         | v7.0.0    | v7.0.0    | 22           | OK

A human reviews and approves before anything gets corrected. No silent overwrites.

Did it work?

Within the first week, the reconciliation pipeline caught something real. Someone had run terraform apply manually — outside the pipeline. The state serial changed, the version tracker didn't. Flagged at 6 AM the next morning.

Separately, the next scheduled deployment caught a portal change. An NSG rule had been modified directly in the Azure Portal. terraform plan showed the unexpected diff and terraform apply reverted it.

Between the two mechanisms, nothing went undetected for more than a day.

Security

No secrets stored anywhere. Everything is OIDC. GitHub App creds, state backend config — all fetched at runtime from Key Vault. Nothing in repo secrets or pipeline variables.

Private endpoints on everything. Storage, Key Vault, AI Search, SQL, PostgreSQL, Container Registry — all deployed with public access disabled.

Quality gates on every PR. Terraform fmt, validate, Checkov, tfsec, commitlint. Doesn't pass, doesn't merge.

Rollback is a first-class thing. Restore any previous version. The tracker records it as intentional so reconciliation doesn't freak out.

Here's the pattern every pipeline job follows:

# Every job:
- name: "Azure Login"
  uses: ./.github/actions/azure-login

- name: "Fetch Secrets from Key Vault"
  id: kv-secrets
  uses: ./.github/actions/keyvault-secrets
  with:
    keyvault_name: ${{ env.KEYVAULT_NAME }}

- name: "Generate GitHub App Token"
  uses: ./.github/actions/github-app-token
  with:
    app_id: ${{ steps.kv-secrets.outputs.app_id }}
    private_key: ${{ steps.kv-secrets.outputs.private_key }}

What we'd tell you

Separate platform from application on day one. The repo boundary is the governance boundary. Shared repo means shared blast radius.

Pin everything. Modules, providers, Terraform versions. If today's deploy can produce a different result tomorrow because something upstream changed, you don't have reproducible infrastructure.

Assume every component will fail on its own. The pipeline will succeed but the registry write will fail. Build verification loops before you need them, not after.

Make the right path the easy path. If uncommenting a Terraform block is easier than writing one from scratch, people will uncomment. If deploying through a form is easier than asking the platform team, they'll use the form. Design for that.

Understand the two types of drift. Pipeline drift (someone ran terraform outside the pipeline) gets caught by metadata comparison. Resource drift (portal changes) gets caught by the next terraform plan. You need both.

The bottom line

We started with: can you prove what's deployed in production right now?

Now we can. And the platform checks it every morning at 6 AM.

Three repos. Five pipelines. 37 modules. SemVer on every deployment. Three tracking files that cross-check each other. One daily alarm clock asking: is everything still true?

It always is. And when it isn't, we know within hours.

If you're managing multi-subscription Azure infrastructure and dealing with drift or visibility problems — we'd genuinely like to hear how you're handling it.

Operating AI Agents on Azure: Observability with Azure AI Foundry

skundapura — Wed, 29 Apr 2026 19:15:11 GMT

Azure AI Foundry is Microsoft’s enterprise platform for building, deploying, and operating AI applications and intelligent agents as first‑class Azure workloads. From an infrastructure perspective, Foundry acts as a control plane that brings together model hosting, agent execution, tooling, evaluation, and observability under a single, governed environment. It integrates natively with Azure services such as Azure Monitor, Application Insights, managed identities, and role‑based access control, allowing AI workloads to be managed with the same rigor as traditional applications. Azure AI Foundry supports both portal‑driven and code‑driven workflows, enabling platform teams to standardize lifecycle management while supporting diverse development styles. This makes Foundry particularly suited for production environments where governance, auditability, and operational consistency are non‑negotiable.

Microsoft Agent Framework (MAF)

Microsoft Agent Framework is an Azure‑native framework designed to build and operate intelligent agents directly within Azure AI Foundry. It treats agents as managed workloads, enabling platform teams to deploy, secure, and monitor them through the Foundry control plane. Tracing is enabled automatically without code changes, capturing agent reasoning, tool calls, and multi‑agent interactions in Application Insights.

From an operational standpoint, MAF offers the strongest alignment with enterprise governance and no‑code observability. SRE teams gain immediate visibility without custom instrumentation, making it the preferred choice where standardization, compliance, and operational simplicity are critical.

Semantic Kernel (SK)

Semantic Kernel provides a structured way to orchestrate LLMs with plugins, planners, and functions in .NET and Python environments. When connected to Azure AI Foundry, it automatically emits telemetry for prompts, responses, function execution, and token usage using Azure inference connectors.

Operationally, Semantic Kernel sits in the low‑code category. It offers strong observability with minimal configuration while allowing teams to retain code‑level control. This makes it suitable for teams that need transparency and structure without fully managed agent hosting.

LangChain

LangChain is a widely adopted open‑source framework for building agent workflows and retrieval‑augmented applications. Azure AI Foundry integrates with LangChain using OpenTelemetry‑based tracers, allowing chain execution, tool calls, and model interactions to be captured in Application Insights.

From a platform perspective, LangChain tracing requires explicit configuration but delivers consistent observability once enabled. It fits organizations that standardize on OSS tooling while still needing enterprise‑grade monitoring and debugging through Azure Monitor.

LangGraph

LangGraph extends LangChain by enabling graph‑based, stateful agent orchestration with conditional routing. Azure AI Foundry traces LangGraph workflows using the same OpenTelemetry integration, capturing node transitions, tool usage, and execution paths.

This model is operationally valuable for complex workflows but introduces higher configuration overhead. It suits advanced teams that need deep insight into non‑linear agent execution while maintaining centralized observability.

OpenAI Agent SDK

The OpenAI Agent SDK provides low‑level control for building custom agent runtimes. Unlike other frameworks, it does not emit telemetry by default. To integrate with Azure AI Foundry, teams must manually instrument OpenTelemetry spans and export them to Application Insights.

This approach offers maximum flexibility but shifts observability responsibility to the engineering team. It is best suited for specialized scenarios where bespoke agent execution outweighs operational simplicity.

Log Tracing in Azure AI Foundry

Log tracing in Azure AI Foundry addresses a core operational gap in agent‑based systems that traditional logging cannot solve. AI agents are non‑linear by nature, often invoking multiple tools, branching based on intermediate reasoning, and coordinating with other agents. Azure AI Foundry uses OpenTelemetry and Azure Monitor Application Insights to capture detailed execution traces that show how requests flow through agents, models, and tools. These traces include timing, errors, token usage, and reasoning paths, enabling teams to debug failures and performance issues with precision. By exposing this telemetry through the Foundry portal and Azure Monitor, platform teams gain a unified view of agent behavior that supports troubleshooting, governance reviews, and production reliability at scale.

Depth of Agent Visibility

Capability	Microsoft Agent Framework (MAF)	Semantic Kernel (SK)	LangChain	LangGraph	OpenAI Agent SDK
Agent decision flow visibility	You can see the full step‑by‑step path of what the agent decided to do, including why it chose a particular action, which step came first, and how it reached the final answer. This is visible automatically in Azure AI Foundry without adding code.	You can see the main execution path of the agent, including which skill or planner was used, but fine‑grained reasoning steps may be abstracted unless you add extra logging.	You usually see the final chain execution and intermediate steps, but the “why” behind decisions is not always clear unless you manually add tracing.	You can see each node in the graph and which path was taken at runtime, making it easier to understand branching decisions.	You mostly see inputs and outputs. Internal reasoning and decision paths are not visible unless you explicitly create and manage custom spans.
Tool invocation tracing	Every tool call is captured automatically, showing which tool was called, with what input, how long it took, and what result it returned, all correlated to the agent run.	Tool and function calls are tracked, but details depend on how plugins are implemented and may need configuration to capture inputs and outputs clearly.	Tool calls are visible when tracing is enabled, but they may appear as generic steps unless you customize the tracer.	Tool usage is clearly tied to graph nodes, making it easier to identify which step called which tool.	Tool calls are only visible if you explicitly instrument them using OpenTelemetry or custom logging.
Multi‑agent support visibility	When multiple agents work together, you can follow a single request as it moves from one agent to another, clearly seeing which agent handled which part of the task.	Multi‑agent flows are possible, but visibility across agents is limited and usually requires manual correlation.	Multi‑agent patterns are supported conceptually, but tracing across agents is not automatic and can be difficult to follow.	Designed for complex workflows, LangGraph allows visibility across multiple agents or components, but still requires configuration.	No built‑in way to trace requests across multiple agents unless you design and implement it yourself.
Automatic span hierarchy	Traces are automatically structured in a parent‑child hierarchy, such as request → agent → decision → tool → model call, without manual setup.	Most spans are structured correctly by default, but complex scenarios may require tuning.	Span hierarchy is available when using the provided Azure tracer, but understanding relationships may require experience.	Hierarchy is preserved across graph nodes, making execution order clearer.	No automatic hierarchy; you must manually create and link spans to understand execution flow.

Tracing Integrations Comparison

Aspect	Microsoft Agent Framework (MAF)	Semantic Kernel (SK)	LangChain	LangGraph	OpenAI Agent SDK
Integration type with Azure AI Foundry	Tracing is built directly into Azure AI Foundry when agents are created or managed in the Foundry portal. The platform automatically enables tracing at the server side.	Tracing works through Azure AI inference connectors that integrate Semantic Kernel with Foundry. Telemetry is emitted automatically once configured.	Tracing is enabled using an OpenTelemetry‑based tracer provided by the langchain-azure-ai package and connected to Foundry.	Uses the same OpenTelemetry tracer as LangChain, extended to support graph‑based execution paths.	No built‑in integration. You must manually add OpenTelemetry instrumentation and export traces to Application Insights.
Code changes required	No code changes are required if you use Foundry‑managed agents. Tracing works out of the box.	Minimal code or configuration is required to attach the Azure AI inference connector and enable telemetry.	Low‑code setup. You must add the tracer dependency and configure environment variables.	Low‑code setup similar to LangChain, plus configuration to trace graph nodes.	Pro‑code. You must explicitly write tracing logic, define spans, and manage exporters yourself.
What tracing actually captures	You can see agent executions, internal decision steps, tool and MCP calls, model requests, token usage, latency, errors, and multi‑agent interactions in one correlated trace.	You can see prompts, responses, function or plugin calls, token usage, latency, and planner execution, depending on configuration.	You see chain execution steps, tool calls, and LLM interactions, but understanding deeper reasoning may require reading the chain logic.	You see each graph node, execution order, conditional branching, and related tool calls, which helps debug complex workflows.	By default you only see inputs and outputs. Detailed decision steps or tool usage appear only if you manually instrument them.
Multi‑agent tracing support	Fully supported. You can follow a single request as it moves between multiple agents inside one trace.	Partially supported. Multi‑agent flows are possible but cross‑agent visibility may need manual correlation.	Limited. Multi‑agent patterns exist, but traces across agents are not clearly linked by default.	Better support than LangChain due to graph structure, but still requires deliberate configuration.	Not supported by default. You must design and implement cross‑agent tracing manually.
Who should use this approach	Platform and infrastructure teams that want enterprise‑grade observability with zero instrumentation effort.	Teams building structured AI workflows in .NET or Python that want good visibility with minimal effort.	Open‑source Python teams that want Azure‑grade observability without leaving LangChain.	Teams building complex, stateful, or branching agent workflows that need visibility into execution paths.	Advanced teams that need full control and are comfortable managing observability pipelines themselves.

For deeper understanding and hands-on implementation, I highly recommend explore this documentation: Observability | Microsoft Learn

Building an Enterprise-Grade SQL Platform on Kubernetes using Crossplane and Azure PostgreSQL

prabhattomar — Wed, 29 Apr 2026 16:58:37 GMT

Strategic Overview

Build a Kubernetes-native SQL platform using Crossplane-based database operator to provision Azure PostgreSQL Flexible Server.
Active–Passive, multi-region database architecture using read replicas and manual promotion for failover.
Private networking, DNS abstraction, and virtual endpoints to ensure secure and stable connectivity.
Azure Traffic Manager + DNS failover strategy to enable global routing and minimize manual intervention.
Enterprise-grade HA/DR, with replication, backup, and failover testing workflows.
Observability via Azure Monitor + Datadog for proactive detection (CPU, replication lag, etc.).
Security-first architecture with private endpoints, Azure AD authentication, and no public access.

Problem Statement

Modern platform teams struggle to offer database-as-a-service (DBaaS) with the same level of automation, governance, and consistency that exists for stateless workloads in Kubernetes.

Key gaps:

Database provisioning is still manual, ticket-driven, or portal-based
Lack of standardized HA/DR patterns across teams
Inconsistent networking, DNS, and security configurations
Failover and DR processes require manual intervention and risk downtime
No unified declarative interface for database lifecycle management

Goals

Design and implement a Kubernetes-native, enterprise-grade SQL platform that:

Exposes databases as declarative Kubernetes resources
Automates provisioning via Crossplane using Azure PostgreSQL Flexible Server
Provides built-in HA/DR capabilities across regions
Enables seamless failover through DNS abstraction
Enforces secure, private, and compliant database access patterns
Integrates observability, backup, and operational controls by default
Delivers a self-service experience for developers without compromising governance

Architecture Overview

Crossplane acts as the control plane, translating Kubernetes intent into Azure-managed DB resources
Azure PostgreSQL Flexible Server provides managed HA + replication primitives
Private DNS + Private Endpoints ensure zero public exposure
Traffic Manager enables global abstraction and failover routing
Replica promotion + DNS switch = DR execution model

Kubernetes-Native Provisioning (Crossplane)

What I built

A custom database resource exposed to Kubernetes users:

kind: XPostgreSQLDatabase

Defines:

Primary and secondary regions
Storage and compute config
Networking + DNS
Security (credentials as secret references)

From config:

Primary region: eastus2
Secondary region: central us
Private DNS zone used: testmulti.postgres.database.azure.com
DB size and storage configured declaratively
Credentials managed via Kubernetes Secret

Crossplane Foundation

Provider configuration uses Azure credentials via Kubernetes secret:

apiVersion: azure.m.upbound.io/v1beta1 kind: ClusterProviderConfig metadata: name: default spec: credentials: source: Secret secretRef: namespace: crossplane-system name: azure-creds key: credentials --- apiVersion: azure.upbound.io/v1beta1 kind: ProviderConfig metadata: name: default spec: credentials: source: Secret secretRef: namespace: crossplane-system name: azure-creds key: credentials

Functions extend composition logic:

apiVersion: pkg.crossplane.io/v1beta1 kind: Function metadata: name: function-patch-and-transform spec: package: xpkg.upbound.io/crossplane-contrib/function-patch-and-transform:v0.8.2

Step-by-Step Implementation

1. Define Platform API

Create XRD (Composite Resource Definition)
Expose database as a Kubernetes primitive (XPostgreSQLDatabase)

2. Build Composition

Map Kubernetes resource → Azure PostgreSQL Flexible Server
Create:
- Primary server
- Replica server (secondary region)
- Networking artifacts (Private Endpoint, DNS)

3. Provision Database

Developer applies custom resource
Crossplane:
- Calls Azure APIs
- Creates full database topology

Control-plane prerequisites

1. Install the Crossplane functions

Your attached functions.yaml installs two functions:

function-patch-and-transform
crossplane-contrib-function-python

apiVersion: pkg.crossplane.io/v1beta1 kind: Function metadata: name: function-patch-and-transform spec: package: xpkg.upbound.io/crossplane-contrib/function-patch-and-transform:v0.8.2 --- apiVersion: pkg.crossplane.io/v1beta1 kind: Function metadata: name: crossplane-contrib-function-python spec: package: ghcr.io/crossplane-contrib/function-python:v0.2.0

2. Configure the Azure provider credentials

Your provider-config.yaml defines both a ClusterProviderConfig and a namespaced ProviderConfig, each reading credentials from the azure-creds secret in the crossplane-system namespace.

apiVersion: azure.m.upbound.io/v1beta1 kind: ClusterProviderConfig metadata: name: default spec: credentials: source: Secret secretRef: namespace: crossplane-system name: azure-creds key: credentials --- apiVersion: azure.upbound.io/v1beta1 kind: ProviderConfig metadata: name: default namespace: crossplane-system spec: credentials: source: Secret secretRef: namespace: crossplane-system name: azure-creds key: credentials

What Crossplane creates from that XR

1. Resource Group

The composition first creates an Azure ResourceGroup, with its location patched from spec.regions.primary.name and its name patched from spec.resourceGroup.name. It also writes the resulting resource group name back into composite status.

- name: resource-group base: apiVersion: azure.upbound.io/v1beta1 kind: ResourceGroup spec: forProvider: location: eastus2 patches: - type: FromCompositeFieldPath fromFieldPath: spec.regions.primary.name toFieldPath: spec.forProvider.location - type: FromCompositeFieldPath fromFieldPath: spec.resourceGroup.name toFieldPath: metadata.name - type: ToCompositeFieldPath fromFieldPath: metadata.name toFieldPath: status.resourceGroupName

2. Backup storage resources

The managed composition also creates:

a storage account with accountReplicationType: GRS
a backup container in that account

- name: backup-storage-account base: apiVersion: storage.azure.upbound.io/v1beta2 kind: Account spec: forProvider: accountTier: Standard accountReplicationType: GRS sharedAccessKeyEnabled: true tags: purpose: postgresql-backups automation: enabled - name: backup-container base: apiVersion: storage.azure.upbound.io/v1beta1 kind: Container

3. Private DNS zone

A PrivateDNSZone is created, and its external name is patched from spec.network.privateDnsZoneName. The zone name is also written back into composite status.

- name: private-dns-zone base: apiVersion: network.azure.upbound.io/v1beta1 kind: PrivateDNSZone metadata: annotations: crossplane.io/external-name: postgres.database.azure.com patches: - type: FromCompositeFieldPath fromFieldPath: spec.network.privateDnsZoneName toFieldPath: metadata.annotations[crossplane.io/external-name] - type: ToCompositeFieldPath fromFieldPath: metadata.annotations[crossplane.io/external-name] toFieldPath: status.dnsZoneName

4. Primary region network

The composition creates a primary virtual network, primary subnet, and a Private DNS zone link. The VNet CIDR comes from spec.regions.primary.cidr. The subnet CIDR is derived from that primary CIDR using a regexp + format transform. The subnet is delegated to Microsoft.DBforPostgreSQL/flexibleServers.

- name: primary-vnet base: apiVersion: network.azure.upbound.io/v1beta1 kind: VirtualNetwork metadata: labels: role: primary spec: forProvider: addressSpace: - 10.0.0.0/16 patches: - type: FromCompositeFieldPath fromFieldPath: spec.regions.primary.cidr toFieldPath: spec.forProvider.addressSpace[0] - name: primary-subnet base: apiVersion: network.azure.upbound.io/v1beta1 kind: Subnet metadata: labels: role: primary spec: forProvider: delegation: - name: fs serviceDelegation: - name: Microsoft.DBforPostgreSQL/flexibleServers

5. Primary database server

The primary Azure PostgreSQL Flexible Server is created with:

private access only (publicNetworkAccessEnabled: false)
subnet delegated from the primary subnet
private DNS zone association
admin credentials patched from the composite spec
SKU, storage, version, retention, and backup settings patched from the composite spec
FQDN and server ID written back into composite status

- name: primary-server base: apiVersion: dbforpostgresql.azure.upbound.io/v1beta1 kind: FlexibleServer metadata: labels: role: primary autoscaling: enabled annotations: management.platform.io/autoscale-enabled: 'true' management.platform.io/backup-enabled: 'true' spec: forProvider: publicNetworkAccessEnabled: false administratorLogin: psqladmin administratorPasswordSecretRef: name: '' namespace: crossplane-system key: password patches: - type: FromCompositeFieldPath fromFieldPath: spec.database.size toFieldPath: spec.forProvider.skuName - type: FromCompositeFieldPath fromFieldPath: spec.database.storageGB toFieldPath: spec.forProvider.storageMb - type: FromCompositeFieldPath fromFieldPath: spec.database.version toFieldPath: spec.forProvider.version - type: FromCompositeFieldPath fromFieldPath: spec.database.backupRetentionDays toFieldPath: spec.forProvider.backupRetentionDays - type: FromCompositeFieldPath fromFieldPath: spec.database.geoRedundantBackup toFieldPath: spec.forProvider.geoRedundantBackupEnabled - type: FromCompositeFieldPath fromFieldPath: spec.security.adminUsername toFieldPath: spec.forProvider.administratorLogin

6. Secondary region network and replica

The composition then creates:

secondary VNet
secondary subnet
DNS link for the secondary VNet
bidirectional VNet peering
a secondary PostgreSQL Flexible Server with createMode: Replica

- name: secondary-server base: apiVersion: dbforpostgresql.azure.upbound.io/v1beta1 kind: FlexibleServer metadata: labels: role: secondary replica: 'true' annotations: management.platform.io/failover-candidate: 'true' management.platform.io/promotion-priority: '1' spec: forProvider: location: centralus createMode: Replica sourceServerId: '' publicNetworkAccessEnabled: false patches: - type: CombineFromComposite combine: variables: - fromFieldPath: metadata.name strategy: string string: fmt: /subscriptions/96618111-38e8-48c0-b564-ee5acde49c15/resourceGroups/postgres-crossplane-rg/providers/Microsoft.DBforPostgreSQL/flexibleServers/%s-primary toFieldPath: spec.forProvider.sourceServerId

7. Read/write DNS records

The composition creates multiple PrivateDNSCNAMERecord resources for read and write endpoint abstraction. These records are patched from spec.network.privateDnsZoneName, spec.network.writeEndpointName, and spec.network.readEndpointName, and some are annotated with management.platform.io/update-on-failover: 'true'.

- name: cname-write base: apiVersion: network.azure.upbound.io/v1beta1 kind: PrivateDNSCNAMERecord metadata: annotations: management.platform.io/managed-by: failover-script management.platform.io/update-on-failover: 'true' spec: forProvider: ttl: 300 - name: cname-read base: apiVersion: network.azure.upbound.io/v1beta1 kind: PrivateDNSCNAMERecord metadata: annotations: management.platform.io/managed-by: failover-script management.platform.io/update-on-failover: 'true' spec: forProvider: ttl: 300

8. Management objects inside Kubernetes

The managed composition also creates Kubernetes-native control objects through the Kubernetes provider:

a ConfigMap for management settings
a ServiceAccount
a ClusterRole
a ClusterRoleBinding

- name: management-config base: apiVersion: kubernetes.crossplane.io/v1alpha1 kind: Object spec: forProvider: manifest: apiVersion: v1 kind: ConfigMap data: backup-enabled: "true" backup-retention-days: "35" autoscaling-enabled: "true" failover-enabled: "true" - name: management-clusterrole base: apiVersion: kubernetes.crossplane.io/v1alpha1 kind: Object spec: forProvider: manifest: apiVersion: rbac.authorization.k8s.io/v1 kind: ClusterRole

High Availability (HA) and Disaster Recovery (DR)

High Availability (HA) Strategy

Design Principles

Ensure minimal disruption for infrastructure-level failures
Leverage managed Azure HA capabilities
Maintain consistent connectivity through private networking

Implementation

1. Zone-Redundant High Availability (Primary Region)

PostgreSQL Flexible Server supports zone-redundant HA deployment
Primary database instance is replicated synchronously across Availability Zones
Platform configuration enables:
- Same-region redundancy
- Automatic failover within region (infrastructure-level issues)

Failover within region is handled by Azure, but cross-region failover is not automatic

2. Resource Connectivity (HA Path)

Within a region, connectivity follows:

Application → Private DNS → Private Endpoint → PostgreSQL Primary

Private endpoints connect database to VNet
Private DNS ensures internal resolution
Traffic never leaves Azure backbone
Public access is disabled

Disaster Recovery (DR) Strategy

Design Principles

Handle regional outages and large-scale failures
Ensure data durability and failover capability
Minimize RPO and RTO impact

1. Cross-Region Replication Architecture

Secondary PostgreSQL server deployed in paired region
Configured as read replica (asynchronous replication)
Example:
- Primary: East US 2
- Secondary: Central US

Primary (Write) ─────────► Replica (Read) Async Replication

Replica is continuously receiving updates but not writable

2. Resource Connectivity (DR Path)

Cross-region setup includes:

Global VNet + Hub-Spoke connectivity
Private endpoints in both regions
Shared Private DNS zone

App → DNS → Traffic Manager → Region Endpoint → Private Endpoint → DB

Cross-region communication uses Azure backbone

3. Failover Process (DR Execution)

Azure PostgreSQL does not provide automatic global failover, hence DR is controlled and explicit

Step-by-step failover:

Detect primary region failure
Promote replica to standalone primary
Update DNS / Traffic Manager routing
Redirect application traffic
Validate connectivity and resume operations

Replica → Promote → Becomes Primary → Traffic redirected

Conclusion

We modeled the database platform as a Kubernetes-native composite API, XPostgreSQLDatabase, and delegated infrastructure realization to a Crossplane pipeline composition. The composition reads user intent from the composite spec—regions, CIDR ranges, DNS settings, database sizing, retention, and admin secret references—and translates that into Azure resources including a resource group, private DNS zone, regional VNets and subnets, bidirectional peering, a primary Azure PostgreSQL Flexible Server, a cross-region replica, and private DNS CNAME records for read/write abstraction. In the managed variant, the composition also creates Kubernetes-side management artifacts for backup, autoscaling, and failover-related configuration.

Migrating Splunk Logs to Azure Application Insights on VMs

skundapura — Wed, 29 Apr 2026 11:35:03 GMT

Phase 0 – Understanding the Current Environment

Before touching any code or agents, map out the existing architecture.

Current setup:

Applications write logs via log4net → local log files on VMs
Splunk Universal Forwarders read logs → send to Splunk Indexers
Splunk dashboards, alerts, and integrations with PagerDuty/ServiceNow
Logic Monitor monitors VMs for CPU, memory, disk, uptime (not logs)

Developer tasks:

Document all log file locations:

grep -r "log4net" /path/to/apps

2. List all Splunk forwarder configurations:

cat /opt/splunkforwarder/etc/system/local/inputs.conf

Record all dashboards, alerts, and their dependencies.
Note any compliance requirements (PHI-sensitive logs, retention policy).

Note: Capture Splunk dashboard screenshot with example alerts and log search query to reference during migration.

Phase 1 – Prepare Azure Environment

Step 1.1 – Create Application Insights and Log Analytics Workspace

Developer Actions:

Go to Azure Portal → Create Resource → Application Insights
Fill in:
- Name: AppName-Insights
- Resource Group: Observability-RG
- Region: closest to your VMs
- Application Type: .NET, Java, or Other
Click Review + Create → Create
Navigate to Log Analytics Workspace
Create workspace for centralizing logs and metrics
- Note Workspace ID and Primary Key for agents

Step 1.2 – Retrieve Instrumentation Key / Connection String

Open Application Insights resource → Properties
Copy Instrumentation Key or Connection String

Note: You will use this for SDK integration later and optional log appender configuration.

Phase 2 – File-Based Ingestion Migration

Since your apps already write logs to files, we can replace Splunk forwarders with Azure Monitor Agent (AMA).

Step 2.1 – Install Azure Monitor Agent (AMA) on VMs

Windows VM:

# PowerShell script to install AMA Install-Agent.ps1 -WorkspaceId "<WorkspaceID>" -WorkspaceKey "<WorkspaceKey>"

Linux VM:

sudo ./install-ama.sh --workspace-id <WorkspaceID> --workspace-key <WorkspaceKey>

Verify installation:

# Windows Get-Service -Name "AzureMonitorAgent" # Linux sudo systemctl status azuremonitoragent

Note: Capture AMA service status to confirm it is running.

Step 2.2 – Configure AMA to Read Log Files

Navigate to Azure Portal → Log Analytics Workspace → Agents Configuration
Add Data Collection Rule:
- Select Custom Logs
- Add path for log4net log files on the VM
- Map fields like timestamp, log level, message
Assign the rule to your VM(s)

Step 2.3 – Validate Log Ingestion

Navigate to Application Insights → Logs
Run a simple query:

traces | order by timestamp desc | limit 50

3. Compare with Splunk logs:

Metric	Splunk Count	Azure Count	Status
Errors last 1h	105	103

Phase 3 – Alert Migration

Step 3.1 – Map Splunk Alerts to Azure Monitor Alerts

Example Mapping:

Splunk Alert	Azure Equivalent	Frequency
Error threshold > 10	Log query alert (`traces	where severityLevel == "Error"

Developer Steps:

Navigate to Azure Monitor → Alerts → + New alert rule
Select Resource → Application Insights → Condition → Custom log search
Configure Action Group:
- PagerDuty, Email, ServiceNow
Set Alert Frequency and Severity

Screenshot Tip: Capture alert creation screen with query and action group.

Phase 4 – Optional SDK-Based Migration (Full Observability)

For deep insights, tracing, and metrics.

Step 4.1 – .NET Applications

Install SDK:

dotnet add package Microsoft.ApplicationInsights.AspNetCore dotnet add package Microsoft.ApplicationInsights.Log4NetAppender

2. Configure in Program.cs:

builder.Services.AddApplicationInsightsTelemetry();

3. Integrate log4net with Application Insights:

4. Add Instrumentation Key in appSettings.json:

{ "ApplicationInsights": { "InstrumentationKey": "YOUR_KEY_HERE" } }

Step 4.2 – Java Applications

Add Maven dependency:

<dependency> <groupId>com.microsoft.azure</groupId> <artifactId>applicationinsights-web</artifactId> <version>3.4.15</version> </dependency>

Configure ApplicationInsights.xml with Instrumentation Key
Enable automatic telemetry: requests, exceptions, dependencies

Step 4.3 – Python Applications

from opencensus.ext.azure.log_exporter import AzureLogHandler import logging logger = logging.getLogger(__name__) logger.addHandler(AzureLogHandler(connection_string='InstrumentationKey=YOUR_KEY')) logger.setLevel(logging.INFO)

Phase 5 – Dual Logging Validation

Keep Splunk forwarders running temporarily
Compare logs and alerts:

Log Type	Splunk	Azure	Status
App Errors	Y	Y	Match
Info Logs	Y	Y	Match

Developer Tip: Fix any missing log parsing or field extraction issues.

Phase 6 – Cutover and Splunk Decommission

Disable Splunk alerts
Gradually stop forwarders on each VM
Archive historical Splunk logs if needed
Remove Splunk agent from VM

Screenshot Tip: Capture AMA and App Insights dashboards fully populated with logs.

Phase 7 – Post-Migration Optimization

Configure Sampling in Application Insights to reduce ingestion cost:

services.Configure<TelemetryConfiguration>(config => { config.DefaultTelemetrySink.TelemetryProcessorChainBuilder .UseSampling(20.0); // 20% sample });

2.Tune dashboards in Azure Workbooks

Setretention and archival policies
Remove unused resources to reduce cost

Phase 8 – PHI Compliance & Security

Avoid logging sensitive PHI in plain text
Use Azure Key Vault for secrets
Enforce RBAC for dashboards and alerts
Enable encryption at rest for logs

Phase 9 – Developer Checklist

Phase	Task	Developer Action
0	Inventory	Document log files, forwarders, dashboards
1	Azure Prep	Create App Insights & Log Analytics
2	File-based ingestion	Install AMA, configure custom logs
3	Alerts	Map and create alerts in Azure Monitor
4	SDK integration	Add AI SDK and log4net appender
5	Validation	Compare Splunk vs Azure logs
6	Cutover	Stop Splunk forwarders, archive logs
7	Optimization	Sampling, retention, dashboards
8	Security	Ensure PHI compliance

Phase 10 – Key Takeaways

Decoupled architecture (log4net → file → Splunk) makes migration simpler
Phase 1: File-based ingestion → minimal code changes, immediate results
Phase 2: SDK instrumentation → full observability (traces, metrics, correlation)
Dual logging is critical for validation
PHI compliance and alert parity must be ensured

Final Developer Tip: Start small with a single service or VM, validate logs and alerts, then scale up to all applications.

Azure Infrastructure Blog articles

CHERIoT-Ibex: Closing the door on memory safety vulnerabilities with hardware-enforced protection

Safely Migrating Terraform Managed Disks on Azure Using Stable Keys and Copilot

The Root Cause: Index-Based "for_each" Keys:

Many Terraform modules flatten VM and disk definitions into a list and use the list index as the for_each key:

for_each = { for index, sp in local.managed_disks : index => sp }

This pattern looks harmless, but the index is not stable:

The result: Terraform plans to destroy and recreate all affected managed disks—even though nothing changed in Azure.

Why This Is Especially Risky on Azure:

Azure managed disks are often:

A forced disk replacement can mean:

This makes state stability a first-class design concern—not an implementation detail.

The Stable Key Pattern:

The fix is conceptually simple: use a domain-stable identifier for each disk.

A proven pattern is:

"${sp.vm}-${sp.data_disk.lun}"

This key is:

Example:

Once applied, adding a new disk results in exactly one new resource, with zero churn.

The Migration Challenge: Terraform State:

Changing for_each keys alone is not enough.

Terraform tracks resources by their state address, not by Azure resource ID. When keys change, Terraform believes:

“The old disks were deleted, and new ones must be created.”

To prevent this, we must move the state, not recreate the resource.

That is where terraform state mv comes in.

Automating the Migration with GitHub Copilot Skills:

To remove risk and human error, the team created a reusable Copilot skill for managed disk key migration.

What the Skill Does:

This ensures:

The skill is stored directly inside the repository under .github/skills, making it:

Example: Generating State Move Commands:

Based on environment JSON, Copilot can generate commands like:

This is repeated deterministically for every existing disk—before any plan or apply.

Recommended Migration Workflow:

Confirm clean state

Update the module

Back up the state

Run terraform state mv

Re-run plan

Add new disks safely

CI/CD and Remote Backend Considerations:

A critical finding from this migration:

terraform state mv always modifies the currently initialized backend.

In pipeline-driven environments:

Failing to align code and state can cause disk destruction in production.

Key Takeaways:

Closing Thoughts:

This pattern is broadly applicable beyond disks—to NICs, extensions, and any resource where identity must outlive ordering.

By combining:

Teams can make infrastructure changes boring again—and that is the ultimate reliability goal.

Building Secure AI Platforms in Banking Using Azure Enterprise Architecture

1. Introduction: AI in Banking Is Not Just a Model Problem

2. Core Design Principle: Controlled Intelligence System

3. Azure Enterprise AI Architecture (Production-Ready Pattern)

4. Private Connectivity Model (Critical for Compliance)

5. Identity-First Security Model (No Secrets Architecture)

6. Secure AI Inference Pipeline

7. RAG Architecture: Enterprise AI Backbone

8. Observability with AMPLS (A Critical Yet Overlooked Layer)

9. Regulatory Mapping: Banking Requirements to Azure Capabilities

10. Real-World Production Challenges

11. Enterprise Architecture Best Practices

12. Azure Service Mapping (Quick Reference)

13. Common Failure Patterns

14. Final Thought

How Validation‑Driven Terraform Made Our Azure Function Deployments Predictable

🚀 Modernizing Azure Landing Zone Deployments: From Terraform Scripts to GHCP-Driven Engineering

Instead of writing infrastructure line by line, we now describe intent — and let AI generate the implementation.

From Infrastructure as Code → Infrastructure by Prompt

How GHCP actually accelerates ALZ deployment

1. Designing the Landing Zone (in minutes, not days)

✅ Output from GHCP

2. Generating Terraform Modules instead of writing them

✅ Output

3. Networking without copy-paste debt

🔹 Prompt

✅ Output

4. OIDC setup without documentation rabbit holes

✅ Output

5. GitHub Actions workflows generated in seconds

7. Agent Design
Step 1: Read Excel Input