The latest Azure HPC SKUs series, HB and HC, unleash the true power of Azure's performant and scalable HPC network interconnect by virtue of Single Root IO Virtualization (SR-IOV). These high performance VM series offer true InfiniBand capabilities and enable all InfiniBand based MPI libraries to run on Azure with full RDMA verbs.
To help HPC users seamlessly take advantage of all the platform features, the Azure HPC team offers a CentOS based HPC image in Azure Marketplace (published by Rogue Wave Software). This image, starting today with CentOS 7.6, includes optimizations and recommended configurations to deliver optimal performance, consistency, and reliability. This is the same image we recently used to set to run the public cloud's first ever 10,000+ core MPI job!
Below is the full list of configuration updates and software installations (on top of a vanilla Azure CentOS VM image):
Configuration Updates:
The included configuration updates are based on optimization recommendations from vendors and partners, as well as learnings from common HPC workloads and usage practices in traditional HPC systems.
- Azure Linux Agent (WAAgent)
- Limit waagent's (VM agent running on every Azure Linux VM) usage of CPU/memory resources.
- Optionally consider disabling waagent at the beginning of your job script, and enabling it back at the end, for CPU sensitive workloads as follows:
sudo systemctl stop waagent <HPC job> sudo systemctl restart waagent
- Higher Memory Limits
- Set max-locked-memory limit to unlimited
- Set number of open files limit to 65535
- Zone Reclaim mode
- Set zone_reclaim_mode to 1
- Disable firewall daemon to help MPI job launchers
- Disable NFS GSS proxy to improve NFS performance
Software Installations:
The CentOS HPC image also comes with a set of pre-installed software packages:
- Mellanox OFED
- Pre-configured IPoIB (IP-over-InfiniBand)
- Development Tools, including GCC (recommended version by platform vendors)
- MPI Libraries and Communication Runtimes
The MPI libraries are configured as environment modules. Users can select any preferred MPI flavor as follows:
module load mpi/<mpi-flavor>
Please refer to the corresponding CentOS-HPC kickstart file for a complete list of configuration updates and software configurations.
Azure HPC team aims to keep the CentOS HPC images up-to-date with latest software versions.
Deploying the CentOS HPC Image
The CentOS HPC images are available from Azure Marketplace, and it can be deployed through a variety of deployment vehicles (CycleCloud, Batch, ARM templates, etc).
Go ahead and give this image a try for your HPC workloads! We're all ears for your feedback, including additional recommendations for optimizations or included software, even other OSes Azure should optimize for HPC purposes.
Note: This image is intended *only* for the SR-IOV enabled Azure instances (such as Azure HB, HC, etc.).
Microsoft
