The numbers show, using continuous integration and continuous deployment (CI/CD) solutions for delivering software and cloud infrastructure improve deployment reliability, recoverability, and agility. In addition to these great benefits, CI/CD produces many byproducts that are also useful. One of these is what I like to refer to as environment traceability.
I like to think of environment traceability as the ability to correlate a deployed application or cloud solution to the individual build that created or updated it. This can be achieved by annotating the deployed objects with information about the build. As an example, perhaps you annotate a cloud object with a tag that includes the CI/CD BuildID.
With these annotations in place, you should be able to correlate a deployment to the build that created it. Once you can do this, you can then trace source control back to the code or infrastructure updates that were made to trigger the build.
In this quick blog post, I will show you how to achieve environment traceability when using Azure Pipelines. The examples given in this blog post are related to Docker container images and Kubernetes, however, the theory can be used on any number of deployment objects.
Built-in Example (Kubernetes Manifest Task)
An Azure Pipeline is essentially a series of automated tasks. Many tasks are built into Azure Pipelines or are available to install from the Visual Studio Marketplace. Some Azure Pipeline tasks include traceability annotations out of the box.
The Kubernetes Manifest task deploys a Kubernetes manifest file to a Kubernetes cluster. When doing so, the objects created by the manifest file are automatically annotated with information about the Azure Pipeline build.
Here is a simple Azure Pipeline that uses the Kubernetes Manifest task to deploy an application to an Azure Kubernetes Service cluster.
trigger:
- master
pool:
vmImage: 'ubuntu-latest'
steps:
- task: KubernetesManifest@0
inputs:
action: 'deploy'
kubernetesServiceConnection: 'blog-kubernetes'
namespace: 'default'
manifests: 'azure-vote.yml'
When running the Pipeline, notice that the pipeline Build ID can be seen in the pipeline address bar.
As a result of running this pipeline, two pods and a Kubernetes service have been created. Let's pull some information about one of the pods using the kubectl describe command. In the following example, I've removed most of the output for brevity.
$ kubectl describe pod azure-vote-back-5966fd4fd4-rxlld
Annotations: azure-pipelines/jobName: "Job"
azure-pipelines/org: https://nepeters-devops.visualstudio.com/
azure-pipelines/pipeline: "neilpeterson.simple-kubernetes-sample (1)"
azure-pipelines/pipelineId: "109"
azure-pipelines/project: kubernetes-deployment-job
azure-pipelines/run: 20191209.5
azure-pipelines/runuri: https://nepeters-devops.visualstudio.com/kubernetes-deployment-job/_build/results?buildId=2206
Note here that the pod was automatically annotated to include information about the build, including the BuildID, which can be matched to the BuildID seen in the previous screenshot.
Custom Example
I think that is pretty awesome, however what if you are not using Kubernetes tasks or another task that has native support for tracing annotations. How can we replicate this behavior in a custom way? The short answer is Predefined Azure Pipeline's variables. Azure Pipelines include a set of internal variables that can be used for various purposes.
For example, let's assume we are using a generic script task to build a Docker container image and that we want to give that image a version that matches the BuildID of the pipeline instance that built the image. For this purpose, we can use a variable named $(Build.BuildId) to return the BuildID.
This example shows the build step, and then also a step that pushes the image to a container registry.
- task: CmdLine@2
displayName: 'docker build (tt-cart)'
inputs:
script: |
docker build -t $(containerRegistryFqdn)/cart.api:$(Build.BuildId) .
- task: CmdLine@2
displayName: 'docker push (tt-web)'
inputs:
script: |
docker push $(containerRegistryFqdn)/cart.api:$(Build.BuildId)
Note here that a container image is created with a version that is equal to $(Build.BuidID).
There are some other steps in my sample pipeline that eventually deploy this container onto a Kubernetes cluster, however, these are not relevant to the post and have been omitted.
Once the pipeline has run, a container image is created with the proper image version, pushed to an Azure Container Registry and then deployed onto a Kubernetes cluster. For this example, the BuildID is 130.
Upon inspecting the container, we can see that the container image version matches the latest pipeline BuildID.
Image: ttacrfqkz63c47hsxc.azurecr.io/cart.api:130
Conclusion
In conclusion, Predefined Azure Pipeline variables can be used to annotate deployment objects in a way that allows us to trace back from production to source code. This can be useful in understanding what has changed in our environment, during troubleshooting, and post-incident review.