Singularity on Azure: Containers for HPC

Azure's customers containerize their most demanding compute workloads with Singularity and execute them on Azure's virtual machines (VMs) especially targeted to HPC and AI. To enable greater integration of Singularity, Azure and Sylabs announced a new collaboration. Singularity container images (Singularity Image Format or SIF files) can now be stored in Azure Container Registry (ACR), and other OCI registries supporting the Open Container Initiative (OCI) Distribution Specification, by leveraging the OCI Registry as Storage (ORAS) project.

“Compliance with standards emerging from the Open Containers Initiative (OCI) has been a matter of emphasis in some of our most-recent releases of Singularity,” stated Singularity founder and Sylabs CEO Gregory Kurtzer. “In fact, Singularity is compliant with both the image and runtime specifications championed by the OCI. To really drive adoption of these standards however, the matter of distributing containers also needs to be addressed. Fortunately, ORAS addresses this significant gap, and significantly lowers the barrier to widespread enterprise adoption. We are delighted to be collaborating on an ongoing basis with Azure to ensure that Singularity is ‘ORAS aware’. Through our initial efforts, SIF container images can now be stored and retrieved in ACR as well as other OCI distribution-based registries. For those seeking to leverage standards-compliant containers in ACR, support for ORAS via Singularity represents a significant advancement.”

Singularity was created as a container solution for scientific and engineering workloads and is amenable to both users and administrators of High Performance Computing (HPC) facilities. It ensures the mobility of applications in a secure way by enabling reproducible, portable and distributable containers with security built in. Tightly integrated with the HPC stack, Singularity supports high speed interconnects like InfiniBand, accelerators like GPUs, resource managers and parallel file systems.

With near bare metal performance, Singularity containers take advantage of the most performant HPC hardware on the cloud such as the H- and N-series VMs on Azure. These are designed to deliver leadership-class performance, scalability and cost efficiency for HPC and AI workloads. The HPC software ecosystem on Azure complements this by supporting Singularity in Batch Shipyard for configuration-based container execution on Azure Batch, CycleCloud to add Singularity to a cluster seamlessly, and recipes to expand re-usability by 'stacking' containers. With this integration of Singularity and ACR, customers now get faster, cheaper access to their container images with geo-replication.

SIF-focused interoperability between Singularity and Azure via ORAS is a work in progress; those interested can monitor and, of course, contribute to the existing collaboration via the project’s GitHub PR. Based upon the rapid rate of progress, users and administrators can anticipate initial availability in an upcoming release of Singularity.

Sylabs and the Singularity community have always been focused on interoperability and this integration extends the concept to create a broader solution-set for the container community. For customers that already use Azure Container Registry or other OCI registries, this new collaboration will allow for an integrated path towards adopting SIF containers into their workflows. It enables customers using Singularity to leverage their investments in ACR and other OCI complaint registries, without having to run and maintain another SIF distribution library. Additionally, customers now have the option of incorporating Azure into their Singularity containerized workflows and can look forward to even closer integration of Singularity across Azure's offerings and services.