Container Image builds on Kubernetes clusters with Containerd and Azure DevOps self-hosted agents

Overview

 

Containerd is the default container runtime with AKS clusters on Kubernetes version 1.19 onwards. With a containerd-based node and node pools, instead of talking to the dockershim, the kubelet will talk directly to containerd via the CRI (container runtime interface) plugin, removing extra hops on the flow when compared to the Docker CRI implementation. As such, you'll see better pod startup latency and less resource (CPU and memory) usage.

 

This change restricts containers from accessing the docker engine, /var/run/docker.sock, or use Docker-in-Docker (DinD).

 

In order to build docker images, Docker-in-Docker is a common technique used with Azure DevOps pipelines running in Self-Hosted agents. With Containerd, the pipelines building docker images no longer work and we need to consider other techniques. This article outlines the steps to modify the pipelines to perform image builds on Containerd enabled Kubernetes clusters.

 

Azure VM scale set agents is an option to scale self-hosted agents outside Kubernetes. To continue running the agents on Kubernetes, we will look at 2 options. One to perform image builds outside the cluster using ACR Tasks and another using kaniko executor image which is responsible for building an image from a Dockerfile and pushing it to a registry.

 

Building images using ACR Tasks

 

ACR Tasks facilitates container image builds.

 

Modify the existing pipelines/create a new pipeline to add an Azure CLI Task running the below command.

 

az acr build --registry <<registryName>> --image <<imageName:tagName>> .

 

The command will:

• Run in the current workspace
• Package the code and upload to in a temp volume attached to ACR Tasks
• Build the container image
• Push the container image to the registry

 

The pipeline should look as illustrated below:

 

 

Though this approach is simple, it has a dependency on ACR. The next option deals with in-cluster builds which does not require ACR.

 

Building images using Kaniko

 

To use Kaniko to build images, it needs a build context and the executor instance to perform the build and push to the registry. Unlike Docker-in-Docker scenario, Kaniko builds are executed in a separate pod. We will use Azure Storage to exchange the context (source code to build) between the agent and the kaniko executor. Below are the steps in the pipeline.

 

• Package the build context as a tar file
• Upload the tar file to Azure Storage
• Create a pod deployment to execute the build
• Wait for the Pod completion to continue

The script to perform the build is as below:

 

# package the source code
tar -czvf /azp/agent/_work/$(Build.BuildId).tar.gz .

#Upload the tar file to Azure Storage
az storage blob upload --account-name codelesslab --account-key $SKEY --container-name kaniko --file /azp/agent/_work/$(Build.BuildId).tar.gz --name $(Build.BuildId).tar.gz

#Create a deployment yaml to create the Kaniko Pod
cat > deploy.yaml <<EOF
apiVersion: v1
kind: Pod
metadata:
  name: kaniko-$(Build.BuildId)
  namespace: kaniko
spec:
  containers:
  - name: kaniko
    image: gcr.io/kaniko-project/executor:latest
    args:
    - "--dockerfile=Dockerfile"
    - "--context=https://<<storageAccountName>>.blob.core.windows.net/<<blobContainerName>>/$(Build.BuildId).tar.gz"
    - "--destination=<<registryName>>/<<imageName>>:k$(Build.BuildId)"
    volumeMounts:
    - name: docker-config
      mountPath: /kaniko/.docker/
    env:
    - name: AZURE_STORAGE_ACCESS_KEY
      value: $SKEY
  restartPolicy: Never
  volumes:
  - name: docker-config
    configMap:
      name: docker-config
EOF

 

The storage access key can be added as an encrypted pipeline variable. Since the encrypted variables are not passed on to the tasks directly, we need to map them to an environment variable.

 

As the build is executed outside the pipeline, it is required to monitor the status of the pod to decide on the next steps within the pipeline. Below is a sample bash script to monitor the pod:

 

# Monitor for Success or failure

while [[ $(kubectl get pods kaniko-$(Build.BuildId) -n kaniko -o jsonpath='{..status.phase}') != "Succeeded" && $(kubectl get pods kaniko-$(Build.BuildId) -n kaniko -o jsonpath='{..status.phase}') != "Failed" ]]; do echo "waiting for pod" && sleep 1; done

# Exit the script with error if build failed

if [ $(kubectl get pods kaniko-$(Build.BuildId) -n kaniko -o jsonpath='{..status.phase}') == "Failed" ]; then 
    exit 1;
fi

 

The complete pipeline should look similar to below:

 

Task 1: [Optional ] Get the KubeConfig (If not supplied through secrets)

 

 

Task 2:  [Optional ] Install Kubectl latest (if not installed with the agent image)

 

Task 3: Package Context and Prepare YAML

Note how the pipeline variable is mapped to the Task Environment variable

 

 

Task 4: Create the Executor Pod

Note: Alternatively, can be included in the script kubectl apply -f deploy.yaml

 

 

Task 5: Monitor for Status

 

 

Summary

 

These build techniques are secure compared to Docker-in-Docker scenario as no special permission, privileges or mounts are required to perform a container image build.