Media and entertainment (M&E) studios often operate on razor-thin margins, so protecting staffing and rendering resources must always be a priority. In light of this, many studios are taking advantage of new tools to automate big render jobs, finding that the better the automation, the more efficiently they can use the cloud and human resources—and ultimately improve their margins.
Let me give you two real-world examples that demonstrate the need for automation. One of our small-studio customers recently told about the high labor and cloud-resources cost of manually setting up their Azure infrastructure (typically a 1,000-node configuration) for rendering workloads. While the use of Azure virtual machine scale sets accelerated node creation, users still had to manually initiate, monitor, and stop the environment—which they often forgot to do, while the meter for cloud services kept running.
At another larger studio, our customer recounted spending an entire day scaling up a 16-node Avere vFXT for Azure cluster. Each node required roughly 20 minutes to start and stabilize. An operator error at the fourteenth node required a restart of the entire process. Clearly each of these studios can benefit from an automated solution that allows render managers to create and destroy infrastructure more easily, reliably, and with many fewer steps.
Use infrastructure as code to automate
New automation tools let you more quickly, reliably, and precisely scale infrastructure up and down so that you’re paying for resources only when you actually need them. These tools work by letting you express your infrastructure as code. Infrastructure as code (IaC) is the management of infrastructure (networks, virtual machines, load balancers, and connection topology) in a descriptive model, using the same versioning as a DevOps team uses for source code. Like the principle that the same source code generates the same binary, an IaC model generates the same environment every time it is applied. IaC allows all infrastructure changes to be captured as code and today is becoming a standard DevOps practice.
Use Terraform to automate, eliminate errors, and reduce costs
Hashicorp Terraform is the IaC application most commonly used application by M&E rendering studios to simultaneously provision and manage both on-premises and cloud infrastructure. It’s an open-source tool (downloadable from Hashicorp) that codifies infrastructure in configuration files that describe the topology of cloud resources. Resources include virtual machines, storage accounts, and networking interfaces. The Terraform command line interface (CLI) provides a simple mechanism to deploy and version the configuration files to Azure.
As an IaC automation tool, Terraform makes it easy to codify configuration processes—in a simple, human-readable form—for provisioning and managing cloud infrastructure. Many in the cloud-burst rendering community are already Terraform users, and they can now use the application to manage Avere vFXT resources. Essentially a template to follow for standard vFXT configurations for rendering, the Terraform tool builds out the infrastructure with just a few commands.
Terraform's template-based configuration files enable you to define, provision, and configure Azure resources in a repeatable and predictable manner. Automating infrastructure, the Terraform tool:
- Deploys the same template multiple times to create identical development, test, and production environments
- Reduces the potential for human errors in deploying and managing infrastructure
- Can automatically destroy/delete the environment, reducing the chance of the studio incurring charges because users forget to stop or delete resources
- Tracks the history of infrastructure changes
- Enables security infrastructure to be described entirely in one place, making it easier to manage, audit, and approve any changes
- Enables decoupling of implementer, approver, and operational roles
- Can be integrated into CI/CD (continuous integration/continuous delivery) pipelines for agility on infrastructure changes (as part of the DevOps mentioned earlier)
- Reduces the cost of development and test environments by creating them on demand (Terraform overview)
In the examples cited earlier, studios can use Terraform IaC to conserve both staffing and rendering sources—in the first case by destroying the environment as soon as the workload completes, and in the second case by eliminating the need for a person to sit in front of a screen for eight hours and by avoiding operator errors.
While many enterprises use Azure Resource Manager (ARM) templates for IaC services, studios that choose to use the Terraform tool have typically already adopted Terraform for their on-premises infrastructure provisioning. Since ARM templates do not work cross platform (for both on-prem and cloud infrastructure), the Terraform application is the better option for these studios.
Use Terraform examples to start deploying Avere vFXT
To help you get started using Terraform for Avere vFXT, we’ve created a set of examples that show you how to deploy HPC Cache, Avere vFXT, and an NFS Filer from minimal to three -node configurations. All examples can be deployed through Azure Cloud Shell. (Terraform is installed by default.) The process follows these four basic steps:
-
Start Azure Shell in browser tab (https://shell.azure.com).
-
Set your subscription.
-
Create your Terraform file from direct editing or by pulling it from a git repository.
-
Run through the processes of:
-
Terraform Init that pulls in the providers (Azure Provider) and any modules.
-
Terraform apply that applies the configuration to production.
-
You will find the below nine examples here.
- No filer example: Shows the steps required to configure a render network, controller, and Avere vFXT without any filers.
- Avere vFXT mounting Azure Blob Storage cloud core filer example: This example configures a render network, controller, and Avere vFXT with an Azure Blob Storage cloud core filer.
- Avere vFXT mounting 1 IaaS NAS filer: Configure a render network, controller, and Avere vFXT with one filer (illustrated below).
- Avere vFXT mounting 3 Iaas NAS filers: Configure a render network, controller, and Avere vFXT with three filers.
-
Avere vFXT for Azure mounting an Azure NetApp volume: This example configures a render network, controller, and Avere vFXT with an Azure NetApp volume.
- Avere vFXT and VDBench: This example shows a basic setup to generate small and medium-sized workloads to test the performance of Avere vFXT for Azure memory and disk subsystems.
- Avere vFXT and VMSS: Provides the basic setup steps to start virtual machine scale sets and round-robin mount Avere vFXT for Azure.
Specialized Avere vFXT for Rendering and Artists:
- Avere vFXT optimized for Houdini: The Avere vFXT in this example has custom settings to optimize the vFXT and the core filer custom configuration for Houdini.
-
Avere vFXT and Cloud Workstations. This example configures a render network, controller, and Avere vFXT with one filer, and lets you choose between Isolated or Shared workstation cache policies
Figure 1. Configuring an Avere vFXT for Azure with one filer
Use these resources and references
The links below provide a wealth of information about getting started using Terraform to provision and manage Avere vFXT for Azure resources. If you need additional information, you can visit our github page or message our team through the tech community.
About Terraform and infrastructure as code
Configuration syntax (Hashicorp configuration language - HCL)
Quickstart: Install and configure Terraform to provision Azure resources
Terraform Examples, Modules, and Providers for HPC Cache and Avere vFXT for Azure