We recently shared information about using Terraform Infrastructure as Code (IaC) for automating render cloud infrastructure deployments. We also provided examples for deploying Avere vFXT for Azure rendering. For many Media & Entertainment (M&E) studios, the next logical step is to integrate their source control management (SCM) system with their Terraform infrastructure deployments via a CI/CD (continuous integration / continuous delivery) pipeline.
Workflow and pipeline management systems, such as GitHub Actions and Azure Pipelines, can effectively enable automated orchestration of build-and-release processes around Terraform infrastructure deployments. In addition, human review and approval controls can also be incorporated before a release. This operational pattern is quite common in M&E studios to enable traceable infrastructure deployments across multiple rendering environments.
In response to recent customer requests, our Azure Media Solutions team has created a sample workflow for managing rendering infrastructure across Azure compute, storage and networking services, including Virtual Networks, Virtual Machine Scale Sets (VMSS), Avere vFXT, and HPC Cache. The deployment automation sample workflow can significantly accelerate rendering, whether you’re an existing Azure customer or deploying rendering for the first time.
By enabling GitHub as the single pane of change management glass for Terraform files under source control, M&E studio IT administrators and rendering teams can reliably and efficiently respond to dynamic business requirements. Managing infrastructure deployments as GitHub pull requests provides several benefits, including: 1) history and audit tracking; 2) versioning and release control; and 3) automated testing and validation. A perfect use case would be M&E studios that need to deploy multiple rendering environments across different geographic regions to support various show productions that are shooting concurrently. In such a high demand setting, GitOps enables efficient and reliable deployment and operational management of the rendering infrastructure.
Before getting into more of the sample solution details, let me start with a short description of GitHub and GitOps for anyone not already familiar with the platform and related software components.
Use GitHub as a source control system
With some 40 million developers currently using GitHub, you or someone on your team has likely already been using GitHub hosting services for source-code control and management. The company offers a number of packages that let you safely publish and consume packages within your organization or with the entire world. Many M&E studios use GitHub Enterprise, a private on-premises GitHub that includes necessary privacy and security controls for locked-down networks.
GitOps are standardized workflows that enable CI/CD—that is, you can automate your delivery pipelines and safely roll out changes to your infrastructure. GitOps enables a ‘source of truth’ for your infrastructure to enable faster, more reliable deployments.
Jumpstart pipeline automation and management using Azure rendering GitOps
In our new GitOps for Azure rendering sample, we use Jenkins software as the CI tool to create the build and release pipelines. Studio rendering teams could also use Azure Pipelines or GitHub Actions for similar functionality. The GitOps workflow also integrates the Terraform files needed to initialize, plan and apply the Azure infrastructure changes (https://techcommunity.microsoft.com/t5/azure-storage/automating-render-cloud-infrastructure/ba-p/127...).
Exclusively enabled via GitHub App development (https://developer.github.com/apps/), pull request code change validation is visualized in the GitOps solution sample via the new GitHub App Checks API (https://developer.github.com/v3/checks/). Both Terraform initialize and plan operations are integrated as Check Runs within a GitHub App pull request. Refer to the following GitHub repository for the sample implementation details (https://github.com/Azure/Avere/tree/master/src/tutorials/GitOps).
Figure 1. Architectural diagram of GitOps for deploying rendering workloads in Azure. The Azure GitOps Workflow process on the bottom right outlines the core sequence of interactions.
Figure 2. GitHub App Pull Request Check Runs for Terraform Init & Plan pipeline stages.
Figure 3. GitHub App Pull Request Comment with Terraform Apply pipeline output.
Resources and references
The links below provide a wealth of information about using GitHub and GitOps. If you need additional information, you can visit our github page or message our team through the tech community.
https://www.weave.works/blog/what-is-gitops-really
https://www.weave.works/technologies/gitops/
https://thenewstack.io/what-is-gitops-and-why-it-might-be-the-next-big-thing-for-devops/
https://enterprise.github.com/faq
https://developer.github.com/apps/
https://developer.github.com/v3/checks/
https://developer.github.com/apps/quickstart-guides/creating-ci-tests-with-the-checks-api/
https://github.com/Azure/Avere/tree/master/src/tutorials/GitOps
https://github.com/features/actions
https://help.github.com/actions