Getting Secrets Out of YAML: Implementing Azure Key Vault CSI Driver on AKS with Workload Identity

Table of Contents

• Why This Pattern Matters
• The Problem with Secrets in YAML
• What We Wanted to Achieve
• Architecture Overview
• How the Flow Works
• Implementation Prerequisites
• Step-by-Step Implementation
• Understanding the YAML Components
• Secret Rotation and Reloaders
• Common Pitfalls and Troubleshooting
• Security and Operational Benefits
• Key Takeaways
• Microsoft Documentation References
• Final Thoughts

Why This Pattern Matters

Most Kubernetes environments start with good intentions around secret management.

Over time, however, many AKS deployments gradually evolve toward patterns like:

value: "#{SomeSecret}#"

A pipeline substitutes the value during deployment, the application works, and the pattern spreads across services.

The problem is that this quietly turns:

• deployment pipelines
• rendered manifests
• release artifacts
• CI/CD logs

into part of the secret distribution path.

Even if the secret originates from Azure Key Vault, the value itself still travels through multiple systems before reaching the pod.

That creates:

• unnecessary exposure risk
• difficult rotation workflows
• broader operational blast radius
• audit complexity

This article walks through a cleaner runtime-based model using:

• Azure Key Vault CSI Driver
• AKS Workload Identity
• Managed Identity federation

where workloads authenticate directly to Key Vault without secret values ever appearing in YAML or pipelines.

The Problem with Secrets in YAML

The traditional pattern usually looks like this:

env: - name: APPLICATION_SECRET value: "#{ApplicationSecret}#"

At deployment time:

1. The pipeline retrieves the secret
2. The placeholder is replaced
3. Kubernetes receives the rendered manifest

Operationally, this means:

• the pipeline temporarily possesses the secret
• the rendered YAML contains the secret
• logs or artifacts may accidentally retain it
• secret rotation requires redeployment

This often exists because:

• Key Vault secret names don't match application configuration names
• teams want backward compatibility
• changing application code is expensive

The Azure Key Vault CSI Driver solves this by introducing a runtime mapping layer instead of a pipeline substitution layer.

What We Wanted to Achieve

The design goals were simple:

• No secret values in YAML
• No secret substitution in CI/CD pipelines
• Runtime-only secret retrieval
• Identity-based authentication
• Support for mapping Key Vault names to application config names
• Minimal or zero application code changes
• Secure secret rotation workflows

Architecture Overview

+----------------------+
|   AKS Application    |
|        Pod           |
+----------+-----------+
           |
           v
+----------------------+
| Kubernetes           |
| ServiceAccount       |
+----------+-----------+
           |
           v
+----------------------+
| AKS Workload         |
| Identity Webhook     |
+----------+-----------+
           |
           v
+----------------------+
| Federated Managed    |
| Identity             |
+----------+-----------+
           |
           v
+----------------------+
| Azure Key Vault      |
+----------+-----------+
           |
           v
+----------------------+
| CSI Driver Mount     |
| (/mnt/secrets-store) |
+----------+-----------+
           |
           v
+----------------------+
| Application Reads    |
| Secret at Runtime    |
+----------------------+

How the Flow Works

When a pod starts:

1. The pod uses a Kubernetes ServiceAccount
2. AKS Workload Identity injects an OIDC token
3. The Secrets Store CSI Driver uses that token
4. Azure validates the federated identity relationship
5. The Managed Identity receives access to Key Vault
6. Secrets are retrieved at runtime
7. Secrets become available

This entire process happens dynamically during pod startup.

No secret values pass through:

• YAML manifests
• pipeline variables
• Helm values
• release artifacts

The pod authenticates as itself.

Implementation Prerequisites

Before implementation, ensure:

Requirement - Purpose

• AKS Cluster - Runtime environment
• Azure Key Vault - Centralized secret store
• OIDC Enabled - Required for federation
• Workload Identity Enabled - Enables identity injection
• Managed Identity - Authentication mechanism
• Secrets Store CSI Driver - Runtime secret retrieval

Implementation

The implementation can generally be divided into two phases:

• Platform Setup - One-time AKS and Azure configuration performed by platform/infrastructure teams
• Application Onboarding - Per-service configuration performed by application teams

This separation is important because most of the complexity exists only once at the platform layer. After the foundational setup is complete, onboarding additional workloads becomes significantly simpler and more repeatable.

The examples below are intended to demonstrate the overall implementation pattern and architecture flow. Exact implementation details may vary depending on:

• organizational RBAC models
• networking restrictions
• Key Vault access configuration
• GitOps/Helm workflows
• cluster governance policies
• AKS versions and add-on configurations

Phase 1 — Platform Setup

1. Enable Required AKS Capabilities

A typical implementation starts by enabling the AKS capabilities required for identity federation and runtime secret retrieval.

These usually include:

• OIDC Issuer - Enables identity federation between AKS and Microsoft Entra ID
• Workload Identity - Injects federated identity tokens into workloads
• Azure Key Vault CSI Driver - Retrieves secrets securely at runtime

Example Azure CLI commands:

az aks update \
  --resource-group <resource-group> \
  --name <aks-cluster> \
  --enable-oidc-issuer \
  --enable-workload-identity

az aks enable-addons \
  --addons azure-keyvault-secrets-provider \
  --resource-group <resource-group> \
  --name <aks-cluster>

Most organizations validate these capabilities before onboarding workloads.

2. Configure a Managed Identity

Each workload or application typically receives its own User-Assigned Managed Identity.

This follows least-privilege principles and helps reduce operational blast radius between services.

Example:

az identity create \
  --name workload-identity \
  --resource-group <resource-group>

The identity is then granted access to retrieve secrets from Azure Key Vault.

Common approaches include:

• Azure RBAC role assignments
• Azure AD group-based access
• Key Vault access policies (legacy environments)

Typical required permissions include:

• Get
• List

for secrets stored in the vault.

3. Configure Federated Identity Trust

Workload Identity relies on federation between:

• AKS
• Kubernetes ServiceAccounts
• Microsoft Entra ID
• Managed Identities

A Federated Identity Credential establishes this trust relationship.

The implementation usually maps:

• a Kubernetes namespace
• a Kubernetes ServiceAccount
• an AKS OIDC issuer

to a specific Managed Identity.

Example structure:

system:serviceaccount:<namespace>:<serviceaccount>

This configuration is one of the most important parts of the setup because federation mismatches are a very common source of authentication failures.

4. Prepare Kubernetes Namespaces and ServiceAccounts

Application namespaces and ServiceAccounts are typically created before workload onboarding begins.

The ServiceAccount acts as the identity boundary for the workload.

Example:

apiVersion: v1
kind: ServiceAccount
metadata:
  name: workload-sa
  namespace: application-namespace
  annotations:
    azure.workload.identity/client-id: "<managed-identity-client-id>"

This annotation links the Kubernetes ServiceAccount to the Azure Managed Identity.

Phase 2 — Application Onboarding

Once the platform capabilities are available, application onboarding becomes significantly simpler.

5. Define Secret Retrieval Using SecretProviderClass

The SecretProviderClass resource defines:

• which Key Vault secrets should be retrieved
• how those secrets should be exposed inside Kubernetes

Example:

apiVersion: secrets-store.csi.x-k8s.io/v1
kind: SecretProviderClass
metadata:
  name: workload-kv-secrets
  namespace: application-namespace

spec:
  provider: azure

  parameters:
    usePodIdentity: "false"
    clientID: "<managed-identity-client-id>"
    keyvaultName: "<keyvault-name>"
    tenantId: "<tenant-id>"

    objects: |
      array:
        - |
          objectName: application-secret
          objectType: secret

  secretObjects:
  - secretName: workload-secret
    type: Opaque
    data:
    - key: APPLICATION_SECRET
      objectName: application-secret

A few important concepts exist here:

• objects - Defines which Key Vault secrets to retrieve
• secretObjects - Optionally syncs secrets into Kubernetes Secrets

This separation allows:

• Key Vault secret naming
• Kubernetes secret naming
• application environment variable naming

to remain independent from one another.

That flexibility is one of the biggest advantages of the CSI Driver approach.

6. Update Workloads to Use Workload Identity

Application deployments are then updated to:

• use the correct ServiceAccount
• enable Workload Identity
• mount the CSI volume
• optionally consume synced Kubernetes Secrets

Typical workload changes include:

Enable Workload Identity

labels:
  azure.workload.identity/use: "true"

Without this label:

• token injection does not occur
• workload federation fails

Attach the ServiceAccount

serviceAccountName: workload-sa

Mount the CSI Volume

volumes:
- name: secrets-store
  csi:
    driver: secrets-store.csi.k8s.io
    readOnly: true
    volumeAttributes:
      secretProviderClass: "workload-kv-secrets"

volumeMounts:
- name: secrets-store
  mountPath: "/mnt/secrets-store"
  readOnly: true

Even if the application ultimately consumes secrets through environment variables, the CSI volume still needs to be mounted because the Kubernetes Secret synchronization occurs only after a successful mount.

Consume Secrets Inside the Application

Applications can consume secrets:

• directly as mounted files
• or through synced Kubernetes Secrets using secretKeyRef

Example:

env:
- name: APPLICATION_SECRET
  valueFrom:
    secretKeyRef:
      name: workload-secret
      key: APPLICATION_SECRET

One of the biggest operational advantages here is that applications usually require little or no code change.

The application still reads:

• environment variables
• configuration values
• mounted files

The underlying secret delivery mechanism changes — not the application contract.

7. Validate the Integration

Once deployed, teams typically validate:

• pod startup success
• successful CSI volume mounts
• secret retrieval from Key Vault
• environment variable injection
• workload authentication

Common validation activities include:

• inspecting mounted secret paths
• checking pod logs
• reviewing CSI Driver logs
• confirming Key Vault access logs
• validating Workload Identity token injection

Most implementation issues generally fall into one of these categories:

• Federated Identity Credential mismatches
• missing workload labels
• Key Vault permission issues
• incorrect ServiceAccount mappings
• missing CSI volume mounts

Operational Recommendation

For production environments, many organizations also add:

• automated secret rotation handling
• restart controllers such as Stakater Reloader
• readiness probes
• rolling deployment strategies
• monitoring and alerting around secret retrieval failures

These additions help ensure secret updates can occur safely with minimal or zero downtime.

Understanding the YAML Components

The implementation consists of three connected resources:

• ServiceAccount - Identity binding
• SecretProviderClass - Secret retrieval definition
• Deployment - Secret consumption

Think of them as a chain:

ServiceAccount
   ↓
Managed Identity
   ↓
SecretProviderClass
   ↓
CSI Driver
   ↓
Deployment

If any naming mismatch exists, secret retrieval fails.

Consistency matters.

Secret Rotation and Reloaders

One major operational advantage is secret rotation.

Previously:

1. Update secret
2. Update pipeline variable
3. Redeploy application

Now:

1. Rotate secret in Key Vault
2. CSI driver refreshes mounted content
3. Application consumes updated value

For applications using environment variables:

• pod restarts are usually required

Many teams use:

• Stakater Reloader

to automatically restart workloads when secrets change.

Combined with:

• multiple replicas
• readiness probes
• rolling deployments

this enables zero-downtime secret refresh workflows.

Common Pitfalls and Troubleshooting

Federated Credential Subject Mismatch

Most common issue.

The subject must exactly match:

system:serviceaccount:<namespace>:<serviceaccount>

Missing Workload Identity Label

This label is mandatory:

azure.workload.identity/use: "true"

Without it:

• no token injection occurs

Missing CSI Volume Mount

Even if using:

secretKeyRef

the CSI volume must still be mounted.

Otherwise:

• Kubernetes Secret sync never occurs

Key Vault Permission Issues

Federation success does not guarantee Key Vault authorization.

The Managed Identity still requires:

• Get
• List

permissions on the vault.

Environment Variables Do Not Auto-Refresh

Secrets mounted as files can refresh dynamically depending on application behavior.

Environment variables do not.

If using:

secretKeyRef

consider:

• Reloader controllers
• rolling restart strategies
• readiness probes

to safely consume rotated secrets.

Security and Operational Benefits

After migration, several improvements become immediately visible:

• No secret values in YAML
• No secret values in pipelines
• Runtime-only secret resolution
• Per-service identity isolation
• Centralized audit visibility in Azure
• Easier secret rotation
• Reduced operational blast radius
• Cleaner DevSecOps posture

Most importantly:

The deployment pipeline stops being part of the secret distribution mechanism.

Key Takeaways

• Secrets should never flow through deployment pipelines
• Workload Identity removes the need for static credentials
• CSI Driver enables runtime secret retrieval directly from Key Vault
• SecretProviderClass allows clean secret name mapping
• File-based secret consumption is more secure than environment variables
• Secret rotation becomes operationally simpler and safer

Microsoft Documentation References

Official Microsoft guidance:

• AKS Workload Identity https://learn.microsoft.com/azure/aks/workload-identity-overview
• Azure Key Vault Provider for Secrets Store CSI Driver https://learn.microsoft.com/azure/aks/csi-secrets-store-driver
• Secrets Store CSI Driver https://secrets-store-csi-driver.sigs.k8s.io/
• Federated Identity Credentials https://learn.microsoft.com/entra/workload-id/workload-identity-federation

These references are extremely useful when troubleshooting federation, RBAC, or CSI mounting issues.

Final Thoughts

The Azure Key Vault CSI Driver + Workload Identity pattern fundamentally changes how secrets flow through AKS environments.

Instead of distributing secrets through:

• YAML
• CI/CD systems
• deployment artifacts

secrets remain protected behind:

• identities
• runtime authorization
• centralized access control

The initial setup is slightly more involved than pipeline substitution, but once the platform foundations are established, onboarding additional workloads becomes lightweight and repeatable.

This is one of the highest-leverage security improvements available for Kubernetes platforms because it removes an entire category of secret exposure risk without requiring major application rewrites.

Secrets belong to identities — not deployment manifests.

#Azure #AKS #Kubernetes #DevSecOps #CloudSecurity #KeyVault #WorkloadIdentity #PlatformEngineering