This article details building and deploying a container to an Azure Kubernetes Service(AKS) cluster in Azure Government cloud using Azure DevOps.
Private AKS Clusters has the API Server accessible only within the virtual network. This limits the deployments from Hosted Azure DevOps agents. To overcome this, a self-hosted agent within the same virtual network needs to be deployed.
Access to Azure Container Registry (ACR) can be restricted to the virtual network using Private Endpoints. This will limit ACR exposure to public internet. Since private ACR is available only within the vnet, self-hosted devops agents comes to the rescue.
Lastly, to ensure that Azure Pipelines can deploy to Azure Government Clouds, Azure Resource Manager Service Connection should be created with an Environment parameter.
The following steps replicates the above setup. The setup has 3 subnets with the following components
Create a Virtual Network and add 3 subnets.
Use the below values for reference:
Address Space: 10.0.0.0/16
Subnet name |
Address Space |
---|---|
acr-snt | 10.0.1.0/24 |
devops-snt | 10.0.2.0/24 |
aks-snt | 10.0.4.0/22 |
az group create --name gov-devops --location usgovtexas
az network vnet create \
--name gov-devops-vnet \
--resource-group gov-devops \
--subnet-name default
az network vnet subnet create \
--address-prefixes 10.0.1.0/24\
--name acr-snt \
--resource-group gov-devops \
--vnet-name gov-devops-vnet
az network vnet subnet create \
--address-prefixes 10.0.2.0/24\
--name devops-snt \
--resource-group gov-devops \
--vnet-name gov-devops-vnet
az network vnet subnet create \
--address-prefixes 10.0.4.0/22\
--name aks-snt \
--resource-group gov-devops \
--vnet-name gov-devops-vnet
Create a Container Registry in the premium tier (required for private link)
az acr create --resource-group gov-devops \
--name mygovacr --sku Premium
Create the registry’s private endpoint in the acr-snt subnet
az network vnet subnet update \
--name acr-snt \
--vnet-name gov-devops-vnet \
--resource-group gov-devops \
--disable-private-endpoint-network-policies
az network private-dns zone create \
--resource-group gov-devops \
--name "privatelink.azurecr.io"
az network private-dns link vnet create \
--resource-group gov-devops \
--zone-name "privatelink.azurecr.io" \
--name govdns \
--virtual-network gov-devops-vnet \
--registration-enabled false
REGISTRY_ID=$(az acr show --name mygovacr \
--query 'id' --output tsv)
az network private-endpoint create \
--name myPrivateEndpoint \
--resource-group gov-devops \
--vnet-name gov-devops-vnet \
--subnet acr-snt \
--private-connection-resource-id $REGISTRY_ID \
--group-id registry \
--connection-name myConnection
NETWORK_INTERFACE_ID=$(az network private-endpoint show \
--name myPrivateEndpoint \
--resource-group gov-devops \
--query 'networkInterfaces[0].id' \
--output tsv)
PRIVATE_IP=$(az resource show \
--ids $NETWORK_INTERFACE_ID \
--api-version 2019-04-01 \
--query 'properties.ipConfigurations[1].properties.privateIPAddress' \
--output tsv)
DATA_ENDPOINT_PRIVATE_IP=$(az resource show \
--ids $NETWORK_INTERFACE_ID \
--api-version 2019-04-01 \
--query 'properties.ipConfigurations[0].properties.privateIPAddress' \
--output tsv)
az network private-dns record-set a create \
--name mygovacr \
--zone-name privatelink.azurecr.io \
--resource-group gov-devops
# Specify registry region in data endpoint name
az network private-dns record-set a create \
--name mygovacr.usgovtexas.data \
--zone-name privatelink.azurecr.io \
--resource-group gov-devops
az network private-dns record-set a add-record \
--record-set-name mygovacr \
--zone-name privatelink.azurecr.io \
--resource-group gov-devops \
--ipv4-address $PRIVATE_IP
# Specify registry region in data endpoint name
az network private-dns record-set a add-record \
--record-set-name mygovacr.usgovtexas.data \
--zone-name privatelink.azurecr.io \
--resource-group gov-devops \
--ipv4-address $DATA_ENDPOINT_PRIVATE_IP
az acr update --name mygovacr --public-network-enabled false
Choose one of the below authentication methods to perform docker login from the DevOps pipelines
az acr update -n mygovacr --admin-enabled true
Note the admin username and password from the ACR Access Keys blade in the portal
Create a private AKS cluster in the aks-snt subnet
Integrate ACR with AKS or create a pull secret
kubectl create secret docker-registry regcred \
--docker-server=<your-registry-server> \
--docker-username=<your-name> \
--docker-password=<your-pword> \
--docker-email=<your-email>
Create a Linux VM in devops-snt subnet to host the DevOps agent.
az vm create \
--resource-group gov-devops \
--name devopsagent \
--image UbuntuLTS \
--admin-username azureuser \
--generate-ssh-keys \
--subnet devops-snt \
--vnet-name gov-devops-vnet
Note the publicIpAddress from the output
Deploy the DevOps agent on the VM
ssh azureuser@publicIpAddress
curl -O <<url copied above>
mkdir myagent && cd myagent
tar zxvf ../{tar file name}
./config.sh
sudo ./svc.sh install
sudo ./svc.sh start
Install docker, az cli and kubectl
sudo apt -y update
sudo apt -y install apt-transport-https ca-certificates curl gnupg-agent software-properties-common
curl -fsSL https://download.docker.com/linux/ubuntu/gpg | sudo apt-key add -
sudo add-apt-repository "deb [arch=amd64] https://download.docker.com/linux/ubuntu $(lsb_release -cs) stable"
sudo apt update
sudo apt install docker-ce docker-ce-cli containerd.io
sudo usermod -aG docker $USER
newgrp docker
curl -sL https://aka.ms/InstallAzureCLIDeb | sudo bash
sudo az aks install-cli
From Azure DevOps, create a Docker Registry Service Connection
To run kubectl task against the AKS cluster, Create Service Principal and grant contributor access to the resource group
From Azure DevOps, create an Azure Resource Manager Service Connection
apiVersion: v1
kind: Pod
metadata:
name: private-reg
spec:
containers:
- name: private-reg-container
image: mygovacr.azurecr.us/nginx-echo-headers:latest
imagePullSecrets:
- name: regcred
trigger:
- master
pool:
name: 'govpool'
steps:
- task: Docker@2
inputs:
containerRegistry: '{Docker Registry Service Connection Name}'
repository: 'nginx-echo-headers'
command: 'buildAndPush'
Dockerfile: 'Dockerfile'
tags: 'latest'
- task: Kubernetes@1
inputs:
connectionType: 'Azure Resource Manager'
azureSubscriptionEndpoint: '{Azure Resource Manager Service Connection Name}'
azureResourceGroup: 'gov-devops'
kubernetesCluster: '{aks cluster name}'
command: 'apply'
arguments: '-f deploy.yaml'
The setup facilitates secure deployments to private ACR/AKS on Azure Government Cloud using Pipelines.
You must be a registered user to add a comment. If you've already registered, sign in. Otherwise, register and sign in.