In this blog we discuss the integration of ArgoCD GitOps tool with Private Azure Kubernetes Cluster (AKS) using Azure Workload Identity
Problem Statement
Think of a scenario where you have one AKS cluster which is going to host ArgoCD itself and few other private AKS clusters where you are going to deploy your application workloads. ArgoCD is supposed to integrate with private workload AKS clusters to be able to connect, authenticate and authorize itself so it can deploy applications to the target private application workload AKS clusters.
Solution
ArgoCD being the CD tool for application deployment, needs to connect to the Target AKS Private cluster where the custom applications of the artifacts are to be deployed. And for a given platform, there could be many such workload clusters where a centralized ArgoCD is supposed to deploy respective workload applications. The Target AKS cluster is private cluster (Control plane is accessible only through private IP) and is configured to rely on Microsoft Entra ID for authentication and Kubernetes RBAC for Authorization.
Since the Target cluster uses Microsoft Entra ID Authentication, ArgoCD should use a service Principal or Managed Identity in order to connect to the target cluster control plane (kube-api server) to synchronize the cluster state for application deployment. Hence, there is an additional configuration to setup a Workload Identity on the AKS cluster where ArgoCD is deployed.
Prepare the ArgoCD AKS cluster
Note: All the steps mentioned in this section are one-time actions to prepare the Argocd aks cluster (in which Argocd is deployed) to connect to target application workloads AKS clusters (where Argocd is supposed to deploy workload applications) which are Microsoft EntraID authentication enabled.
Below we discuss the tasks/steps required to configure this ArgoCD based solution.
Note : in the example commands below, argocd-aks is the name of AKS cluster where ArgoCD is deployed. argocd-rg is the resource group in which argocd-aks is provisioned.
1. Enable OIDC (OpenID connect) and Workload Identity on the argocd aks cluster
# CLI Command
az aks update --resource-group <resource-group-name> --name <cluster-name> --enable-oidc-issuer --enable-workload-identity
E.g
az aks update --resource-group argocd-rg --name argocd-aks --enable-oidc-issuer --enable-workload-identity
2. Get the OIDC issuer URL and make a note of it to be used later.
# Get OIDC issuer URL from argocd aks cluster
# CLI Command
az aks show --subscription <azure-subscription-name> -g <resource-group-name> --name "<cluster-name>" --query 'oidcIssuerProfile.issuerUrl' -otsv
# Example
az aks show --subscription XYZ_Dev -g argocd-rg --name "argocd-aks" --query 'oidcIssuerProfile.issuerUrl' -otsv
# Returns something like below OIDC url
https://westeurope.oic.prod-aks.azure.com/12345XXXXXXXXXXXXX89/567dYYYYYYYYYYf89/
3. Create a user-assigned managed identity for the argocd aks cluster
# CLI Command
az identity create -g <resource-group-name> -n <user-assigned-identity-name>
# Example
az identity create -g argocd-rg -n argocd-workload-identity
4. Assign the managed identity to the argocd-aks cluster
# Assign managed identities to cluster
# Template
az aks update --resource-group <resource-group-name> --name <cluster-name> --enable-managed-identity --assign-identity <identity-resource-id>
# Example
az aks update --resource-group argocd-rg --name argocd-aks --enable-managed-identity --assign-identity "/subscriptions/XYZ_Dev/resourcegroups/argocd-rg/providers/Microsoft.ManagedIdentity/userAssignedIdentities/argocd-workload-identity"
5. Created a federated credential for the argocd aks cluster
Kubernetes workload (ArgoCD in our case) will use a custom service account to exchange a kubernetes token in favor of an Azure AD token. Kubernetes workload uses a Kubernetes service account which should be bound to the federated credential of the Managed Identity that was created and assigned to argocd-aks cluster in the previous steps.
The following diagrams illustrate how a token exchange works and how this concept of workload identity can be used by AKS clusters to access other azure resources/services.
To create federated credential,
# Create a federated credential
# CLI Command
az identity federated-credential create --name <federated-credential-name> \
--identity-name <user-assigned-identity-name> --resource-group <resource-group-name>
--issuer <oidc-issuer-url-from-argocd-cluster> \
--subject system:serviceaccount:<kubernertes-namespace>:<kubernetes-service-account-name>
# Example
az identity federated-credential create --name argocd-aks-federated-cred \
--identity-name argocd-workload-identity --resource-group argocd-rg
--issuer https://westeurope.oic.prod-aks.azure.com/12345XXXXXXXXXXXXX89/567dYYYYYYYYYYf89/ \
--subject system:serviceaccount:argocd:svc-argocd-aks
6. Create a Kubernetes Service Account to associate with the federated credential.
In the previous step where federated credential is created, subject is referred as --subject system:serviceaccount:argocd:svc-argocd-aks. It means that the service account with name svc-argocd-aks is to be created in the namespace argocd.
# create Service Account with name as "svc-argocd-aks" in the namespace argocd
# 456XXXXXXXXXXXX123 & 789YYYYYYYYY654 in the below ServiceAccount spec refer to the clientId of the managed Identity and the Azure AD tenant ID where the managed identity is created in earlier steps.
# use this command "az identity show --name <user-assigned-identity-name> --resource-group <resource-group-name>" to find the clientId and Tenant ID of the managed identity
apiVersion: v1
kind: ServiceAccount
metadata:
name: svc-argocd-aks # Service account name that we put in --subject
namespace: argocd
#
# Annotate the service account with the Azure AD application which
# has access to Control plane cluster.
annotations:
azure.workload.identity/client-id: 456XXXXXXXXXXXX123
azure.workload.identity/tenant-id: 789YYYYYYYYYYYY654
labels:
azure.workload.identity/use: "true"
# Use Kubectl to apply this manifest
kubectl apply -f svc.yaml # svc.yaml contains the YAML values above
We can say that the Kubernetes service account used by the Kubernetes argocd workload is bound to the federated credential of the Managed Identity.
Assign Cluster-admin role to the Service Account created in above step
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
labels:
addonmanager.kubernetes.io/mode: Reconcile
kubernetes.io/cluster-service: "true"
name: argocd-cluster-serviceaccount
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: ClusterRole
name: cluster-admin
subjects:
- kind: ServiceAccount
name: svc-argocd-aks
namespace: argocd
# Use Kubectl to apply this manifest
kubectl apply -f svc-clusteradmin.yaml # svc-clusteradmin.yaml contains the YAML values above
8. Configure the deployment of ArgoCD
Edit (overwrite) the deployment manifest of argocd-server and the statefulset of argocd-application-controller by mounting a projection volume for your service account token
- Mount a projection volume for the service account token
volumes:
- name: azure-identity-token
projected:
defaultMode: 420
sources:
- serviceAccountToken:
audience: api://AzureADTokenExchange
expirationSeconds: 3600
path: azure-identity-token
serviceAccountName: svc-argocd-aks
- Add some environment variables to tell ArgoCD to use Workload Identity
env:
- name: AZURE_CLIENT_ID
value: <clientId of the workload managed identity>
- name: AZURE_TENANT_ID
value: <Azure AD tenant ID where the workload managed identity is created>
- name: AZURE_FEDERATED_TOKEN_FILE
value: /var/run/secrets/tokens/azure-identity-token
- name: AZURE_AUTHORITY_HOST
value: https://login.microsoftonline.com/
Your final argocd-server deployment file and argocd-application-controller statefulset may look something like below :
# argocd-application-controller statefulset
---
apiVersion: apps/v1
kind: StatefulSet
metadata:
name: argocd-application-controller
spec:
template:
spec:
volumes:
- name: azure-identity-token
projected:
defaultMode: 420
sources:
- serviceAccountToken:
audience: api://AzureADTokenExchange ## possibly this will be different for Azure Gov Cloud, need to check
expirationSeconds: 3600
path: azure-identity-token
serviceAccountName: svc-argocd-aks
containers:
- name: argocd-application-controller
env:
- name: AZURE_CLIENT_ID
value: 456XXXXXXXXXXXX123
- name: AZURE_TENANT_ID
value: 789YYYYYYYYY654
- name: AZURE_FEDERATED_TOKEN_FILE
value: /var/run/secrets/tokens/azure-identity-token
- name: AZURE_AUTHORITY_HOST
value: https://login.microsoftonline.com/
volumeMounts:
- mountPath: /var/run/secrets/tokens
name: azure-identity-token
readOnly: true
# argocd-server deployment
---
apiVersion: apps/v1
kind: Deployment
metadata:
name: argocd-server
spec:
template:
spec:
volumes:
- name: azure-identity-token
projected:
defaultMode: 420
sources:
- serviceAccountToken:
audience: api://AzureADTokenExchange
expirationSeconds: 3600
path: azure-identity-token
serviceAccountName: svc-argocd-tools-cluster
containers:
- name: argocd-server
env:
- name: AZURE_CLIENT_ID
value: 456XXXXXXXXXXXX123
- name: AZURE_TENANT_ID
value: 789YYYYYYYYY654
- name: AZURE_FEDERATED_TOKEN_FILE
value: /var/run/secrets/tokens/azure-identity-token
- name: AZURE_AUTHORITY_HOST
value: https://login.microsoftonline.com/
volumeMounts:
- mountPath: /var/run/secrets/tokens
name: azure-identity-token
readOnly: true
Apply the updated deployment manifest and statefulset manifest to the argocd cluster
Add the target workloads AKS Cluster to the ArgoCD
Note: steps in this section are to be repeated for every target workload AKS cluster to which the ArgoCD is supposed to deploy the workloads
While Argocd UI and also argocd cli commands provides an easy way to add target workloads aks cluster (where argocd is supposed to deploy applications) to argocd , it gets tricky when the target aks cluster is private and also Microsoft EntraID authentication is enabled.
In terms of the enabling network connectivity between Argocd aks cluster and the target private aks cluster, refer this page.
The following steps must be executed for each target workload cluster that needs addition to the argocd.
- Create the ArgoCD secret (on the argocd cluster) to add a target AKS cluster
ArgoCD uses a secret to store all of the information in order to connect to an external cluster.
Sample secret is provided below
# devworkloadaks1 is target workload aks cluster to be added to argocd
# https://dev-abcdxyz.123*******456.privatelink.westeurope.azmk8s.io:443 is the url of control plane of the target aks cluster accessible to the vnet in which argocd aks is deployed.
# 456XXXXXXXXXXXX123 is the client id of the workload managed identity
# 789YYYYYYYYY654 is the azure ad tenant id in which the managed identity is created
# /var/run/secrets/tokens/azure-identity-token is the mount volume where the exchanged token are stored
# caData: <base64 encoded certificate>, this is the certificate authority data encoded in base64 from the kubeconfig file when connected to the control plane of target aks cluster
# the secret is to be created on the argocd cluster
apiVersion: v1
kind: Secret
metadata:
name: devworkloadaks1-secret
namespace: argocd
labels:
argocd.argoproj.io/secret-type: cluster
type: Opaque
stringData:
name: devworkloadaks1
server: https://dev-abcdxyz.123*******456.privatelink.westeurope.azmk8s.io:443
config: |
{
"execProviderConfig": {
"command": "argocd-k8s-auth",
"env": {
"AAD_ENVIRONMENT_NAME": "AzurePublicCloud", ## possibly this will be different for Azure Gov Cloud
"AZURE_CLIENT_ID": "456XXXXXXXXXXXX123",
"AZURE_TENANT_ID": "789YYYYYYYYY654",
"AZURE_FEDERATED_TOKEN_FILE": "/var/run/secrets/tokens/azure-identity-token",
"AZURE_AUTHORITY_HOST": "https://login.microsoftonline.com/",
"AAD_LOGIN_METHOD": "workloadidentity"
},
"args": ["azure"],
"apiVersion": "client.authentication.k8s.io/v1beta1"
},
"tlsClientConfig": {
"insecure": false,
"caData": "LS0tLS1XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX""
}
}
# Use Kubectl to apply this manifest to the argocd cluster
kubectl apply -f secret-target.yaml # secret-target.yaml contains the YAML values above.
2. Finally, assign a Cluster admin role to the argocd workload identity on the target workload aks cluster. This will allow the managed identity to be able to manage the target workload aks cluster where workload applications are to be deployed. Create this ClusterRoleBinding on the target workload aks cluster.
Note: you can further fine tune the permissions as needed to align with the principle of least-privileges
# 456***abc*******xyz789 is the objectId of the managed identity - argocd-workload-identity
# use below command to find the managed identiy of the argocd-workload-identity...
# az aks show --name <user-assigned-identity-name> --resource-group <resource-group-name> --query "identity.userAssignedIdentities"
# Create this ClusterRoleBinding on the target workload aks cluster.
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
labels:
addonmanager.kubernetes.io/mode: Reconcile
kubernetes.io/cluster-service: "true"
name: aks-cluster-admin-federated
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: ClusterRole
name: cluster-admin
subjects:
- apiGroup: rbac.authorization.k8s.io
kind: User
name: 456***abc*******xyz789
# Use Kubectl to apply this manifest on the target workload aks cluster
kubectl apply -f mi-clusteradmin.yaml # mi-clusteradmin.yaml contains the YAML values above
Conclusion
You should now be able to see the target workload AKS cluster added to the ArgoCD UI
Microsoft