Accelerating OpenFOAM Integration with Azure CycleCloud

Microsoft

Feb 13, 2024

OpenFOAM, which stands for Open Field Operation and Manipulation, is a free, open-source software framework primarily used for computational fluid dynamics (CFD) simulations. It provides a wide range of numerical methods and solvers to simulate fluid flow, heat transfer, chemical reactions, and other related phenomena.

Developed by the OpenFOAM Foundation, OpenFOAM is written in C++ and includes a comprehensive set of libraries and tools for simulating complex fluid flows in various engineering and scientific applications. It supports various mesh types, including structured, unstructured, and Cartesian meshes, and offers parallel computing capabilities to leverage multicore processors and high-performance computing clusters.

OpenFOAM is widely used in automotive, aerospace, energy, and environmental engineering industries for tasks like aerodynamics analysis, combustion modelling, multiphase flow simulations, and more. Its open-source nature allows users to customize and extend its functionality according to their specific needs, making it a popular choice for academic research and industrial applications.

Azure CycleCloud offers effective and efficient solutions for running OpenFOAM simulations, whether for individuals or organizations. In this particular scenario, we'll explore a Slurm-based high-performance computing (HPC) setup utilizing HBv4 virtual machines and Ubuntu 22.04.

The following table shows the components used in the demonstration.

CycleCloud	8.5.0-3196
Slurm	23.02.6-1
OS	microsoft-dsvm:ubuntu-hpc:2204:latest
OpenFOAM	v2312
MPI	Open MPI 4.1.5rc2 (Nvidia HPC-X v2.16)
Compute VM SKU	Standard_HB176-96rs_v4 (96 AMD EPYC™ 9V33X ("Genoa-X") CPU cores with AMD's 3D V-Cache, clock frequencies up to 3.7 GHz, 768 GB of RAM, 2.3 GB L3 cache, 400 Gb/s NDR InfiniBand
Scheduler VM SKU	Standard_D4ads_v5

CycleCloud Configuration

To create a Slurm Cluster, you must first ensure that CycleCloud is configured and running. Then, using the CycleCloud GUI, follow the instructions in the Create an HPC cluster from built-in templates guide to create a basic Slurm Cluster.

During the cluster creation process, most parameters can be left at their default values. However, there are a few that need to be updated based on your specific requirements. Under the "Required Settings" section of the Slurm Cluster creation screen, select the appropriate VM SKU for computation. For this demo, we are using Standard_D4ads_v5 for the scheduler and Standard_HB176-96rs_v4 as the HPC VM Type for compute. Additionally, set the Max HPC Cores to 192, as we will be running a 2 Node (2 x 96) MPI job with Infiniband in this test. If you are using more VMs, update the Max HPC cores accordingly.

In the “Advanced Settings” under “Slurm Settings” select 23.02.6-1 and in the “Software”, section select Ubuntu 22.04 LTS for all Machines.

We'll utilize CycleCloud's Cloud-Init feature to automate the installation of required dependencies for setting up OpenFOAM on Ubuntu 22.04. This script streamlines the installation process, guaranteeing that all necessary dependencies are installed accurately.

#cloud-config
runcmd:
  - wget https://raw.githubusercontent.com/vinil-v/of2312-build-repo/main/install_deps_of2312.sh
  - bash install_deps_of2312.sh

Save all the configurations and start the cluster.

Download and Compile OpenFOAM v2312 on Ubuntu 22.04

Engineers frequently prefer compiled versions of OpenFOAM over pre-compiled ones due to various benefits. These include the ability to optimize performance, access cutting-edge features, customize functionality, manage dependencies, optimize for HPC environments, and support educational and research endeavours. By compiling OpenFOAM from source, engineers can personalize the software to suit their precise requirements, leading to superior simulation outcomes. This customization ultimately drives advancements in computational fluid dynamics, facilitating groundbreaking discoveries in fluid dynamics and engineering applications.

To commence building OpenFOAM-v2312, access the Scheduler VM and clone the of2312-build-repo with the command:

git clone https://github.com/vinil-v/of2312-build-repo.git

This step is crucial for accessing the necessary files and initiating the building process.

You will find 4 scripts under of2312-build-repo

vinil@ofcls-prod-scheduler:~$ cd of2312-build-repo/
vinil@ofcls-prod-scheduler:~/of2312-build-repo$ ls -l
total 20
-rw-rw-r-- 1 vinil vinil   65 Feb 12 15:47 README.md
-rw-rw-r-- 1 vinil vinil 2427 Feb 12 15:47 build_of2312-job.sh
-rw-rw-r-- 1 vinil vinil 1444 Feb 12 15:47 download_of2312.sh
-rw-rw-r-- 1 vinil vinil 1746 Feb 12 15:47 install_deps_of2312.sh
-rw-rw-r-- 1 vinil vinil 2755 Feb 12 15:47 motorbike_sim_job_of2312.sh

download_of2312.sh: This script is responsible for downloading OpenFOAM-v2312 and extracting its contents. It automates the process of acquiring the necessary files, ensuring that they are readily available for the subsequent steps in the installation or compilation process.

build_of2312-job.sh: This script facilitates the compilation of OpenFOAM-v2312 through a Slurm job. It sets up the necessary environment and configurations required for the compilation process, ensuring that it can be executed efficiently within a distributed computing environment managed by Slurm.

motorbike_sim_job_of2312.sh: This script serves as a test job for running an OpenFOAM simulation using OpenFOAM-v2312. It provides a sample scenario or simulation setup, such as simulating a motorbike, to demonstrate the functionality and capabilities of the OpenFOAM software. This allows users to verify that the installation and compilation of OpenFOAM-v2312 was successful and that the software is functioning as expected for running simulations.
install_deps_of2312.sh: Installs dependencies required for OpenFOAM-v2312 installation. This is used via cloud-init in cyclecloud.

Run the download_of2312.sh file to download and extract the OpenFOAM-v2312 version.

vinil@ofcls-prod-scheduler:~/of2312-build-repo$ bash download_of2312.sh
Downloading OpenFOAM v2312...
Downloading ThirdParty package...
Extracting OpenFOAM v2312...
Extracting ThirdParty package...
Download and extraction completed.

The OpenFOAM compilation process on a 4-core node takes more than 5 hours, which is considerably long. Therefore, I've chosen not to run the compilation on a Scheduler node due to its limited resources and long compilation time. To speed up the compilation process, I've opted to run it as a Slurm job instead. Slurm will allocate a Standard_HB176-96rs_v4 node with 96 cores for this purpose. With this setup, the compilation job is completed in less than 20 minutes, significantly reducing the overall turnaround time and improving productivity. You could check the OpenFOAMv2312-Build-Job_<ID>.out file to check the progress of the compilation process.

Submit the build_of2312-job.sh using sbatch command to start the compilation process.

vinil@ofcls-prod-scheduler:~/of2312-build-repo$ sbatch build_of2312-job.sh
Submitted batch job 2
vinil@ofcls-prod-scheduler:~/of2312-build-repo$ squeue
             JOBID PARTITION     NAME     USER ST       TIME  NODES NODELIST(REASON)
                 2       hpc OpenFOAM    vinil CF       0:04      1 ofcls-prod-hpc-1

Here is the Compilation slurm job script for the reference:

#!/bin/bash
# SLURM Directives
#SBATCH --job-name=OpenFOAMv2312-Build-Job
#SBATCH --ntasks=96
#SBATCH --output=%x_%j.out
#SBATCH --partition=hpc

# Load MPI and Prepare Software
module purge
module load mpi/hpcx
# Software Installation and Configuration
export INSTALL_DIR="$HOME/OpenFOAM"
# Compiling OpenFOAM
# Load OpenFOAM environment
. "$INSTALL_DIR/OpenFOAM-v2312/etc/bashrc"
cd "$INSTALL_DIR/OpenFOAM-v2312/"

# Set the number of compilation processes
export WM_NCOMPPROCS=$SLURM_NTASKS

# Start compilation
echo "Starting the compilation job at $(date)" 
./Allwmake -j -s -q -l
echo "Finishing the compilation job at $(date)" 
# Post-compilation Check
foamInstallationTest

Check the OpenFOAMv2312-Build-Job_<ID>.out log file for the Compilation job for any error. At the end of the job, it checks for foamInstallationTest to verify the compilation job.

vinil@ofcls-prod-scheduler:~/of2312-build-repo$ tail -60 OpenFOAMv2312-Build-Job_2.out
Basic setup :
-------------------------------------------------------------------------------
OpenFOAM:            OpenFOAM-v2312
ThirdParty:          ThirdParty-v2312
Shell:               bash
Host:                ofcls-prod-hpc-1
OS:                  Linux version 5.15.0-1041-azure
-------------------------------------------------------------------------------

Main OpenFOAM env variables :
-------------------------------------------------------------------------------
Environment           FileOrDirectory                          Valid      Crit
-------------------------------------------------------------------------------
$WM_PROJECT_USER_DIR  /shared/home/vinil/OpenFOAM/vinil-v2312   no        no
$WM_THIRD_PARTY_DIR   .../home/vinil/OpenFOAM/ThirdParty-v2312  yes       maybe
$WM_PROJECT_SITE      [env variable unset]                                no
-------------------------------------------------------------------------------

OpenFOAM env variables in PATH :
-------------------------------------------------------------------------------
Environment           FileOrDirectory                          Valid Path Crit
-------------------------------------------------------------------------------
$WM_PROJECT_DIR       ...ed/home/vinil/OpenFOAM/OpenFOAM-v2312  yes  yes  yes

$FOAM_APPBIN          ...12/platforms/linux64GccDPInt32Opt/bin  yes  yes  yes
$FOAM_SITE_APPBIN     ...12/platforms/linux64GccDPInt32Opt/bin  no        no
$FOAM_USER_APPBIN     ...12/platforms/linux64GccDPInt32Opt/bin  no        no
$WM_DIR               ...e/vinil/OpenFOAM/OpenFOAM-v2312/wmake  yes  yes  often
-------------------------------------------------------------------------------

OpenFOAM env variables in LD_LIBRARY_PATH :
-------------------------------------------------------------------------------
Environment           FileOrDirectory                          Valid Path Crit
-------------------------------------------------------------------------------
$FOAM_LIBBIN          ...12/platforms/linux64GccDPInt32Opt/lib  yes  yes  yes
$FOAM_SITE_LIBBIN     ...12/platforms/linux64GccDPInt32Opt/lib  no        no
$FOAM_USER_LIBBIN     ...12/platforms/linux64GccDPInt32Opt/lib  no        no
$FOAM_EXT_LIBBIN      ...v2312/platforms/linux64GccDPInt32/lib  yes  yes  maybe
$MPI_ARCH_PATH        ...-cuda12-gdrcopy2-nccl2.18-x86_64/ompi  yes  yes  yes
-------------------------------------------------------------------------------

Software Components
-------------------------------------------------------------------------------
Software     Version    Location
-------------------------------------------------------------------------------
flex         2.6.4      /usr/bin/flex
make         4.3        /usr/bin/make
wmake        2312       /shared/home/vinil/OpenFOAM/OpenFOAM-v2312/wmake/wmake
gcc          11.4.0     /usr/bin/gcc
g++          11.4.0     /usr/bin/g++
-------------------------------------------------------------------------------
icoFoam      exists     ...OAM-v2312/platforms/linux64GccDPInt32Opt/bin/icoFoam

Summary
-------------------------------------------------------------------------------
Base configuration ok.
Critical systems ok.

Done

Submitting OpenFOAM Motorbike example job

This job serves as a test job for running an OpenFOAM simulation using OpenFOAM-v2312. It provides a sample scenario or simulation setup, such as simulating a motorbike, to demonstrate the functionality and capabilities of the OpenFOAM software. This allows users to verify that the installation and compilation of OpenFOAM-v2312 was successful and that the software is functioning as expected for running simulations.

Submit the motorbike_sim_job_of2312.sh using sbatch to run the job in 2 x Standard_HB176-96rs_v4 with 192 CPUs.

Job submission script:

#!/bin/bash
# SLURM Directives
#SBATCH --job-name=OpenFOAM-MotorBike-Sim
#SBATCH --ntasks=192
#SBATCH --output=%x_%j.out
#SBATCH --partition=hpc

# Load MPI and OpenFOAM Environment
module purge
module load mpi/hpcx

export INSTALL_DIR="$HOME/OpenFOAM"
. "$INSTALL_DIR/OpenFOAM-v2312/etc/bashrc"

# Nodefile Generation
scontrol show hostname "$SLURM_JOB_NODELIST" | sort -u > "$HOME/nodefile-$SLURM_JOB_ID"

# Case Setup
mkdir "$HOME/run"
cp -r "$FOAM_TUTORIALS/incompressible/simpleFoam/motorBike" "$HOME/run"
cd "$HOME/run/motorBike"

# Parameter Setup and Simulation Run for 192 CPUs
cp "system/decomposeParDict.6" "system/decomposeParDict.$SLURM_NTASKS"
sed -i "s/numberOfSubdomains.*/numberOfSubdomains $SLURM_NTASKS;/" "system/decomposeParDict.$SLURM_NTASKS"
sed -i "s/ n           (3 2 1)/ n           (48 4 1);/" "system/decomposeParDict.$SLURM_NTASKS"
sed -i "s/(20 8 8)/(100 40 40)/" "system/blockMeshDict"
sed -i "s#system/decomposeParDict.6#system/decomposeParDict.$SLURM_NTASKS#" "Allrun"
mpiopts="$mpiopts -np $SLURM_NTASKS --hostfile $HOME/nodefile-$SLURM_JOB_ID"
./Allrun

Submit the job and check the output file for details.

vinil@ofcls-prod-scheduler:~/of2312-build-repo$ sbatch motorbike_sim_job_of2312.sh
Submitted batch job 3


vinil@ofcls-prod-scheduler:~/of2312-build-repo$ cat OpenFOAM-MotorBike-Sim_3.out
Loading mpi/hpcx
  Loading requirement:
    /opt/hpcx-v2.16-gcc-mlnx_ofed-ubuntu22.04-cuda12-gdrcopy2-nccl2.18-x86_64/modulefiles/hpcx
Running surfaceFeatureExtract on /shared/home/vinil/run/motorBike
Running blockMesh on /shared/home/vinil/run/motorBike
Running decomposePar on /shared/home/vinil/run/motorBike
Running snappyHexMesh (192 processes) on /shared/home/vinil/run/motorBike
Running topoSet (192 processes) on /shared/home/vinil/run/motorBike
Restore 0/ from 0.orig/  [processor directories]
Running patchSummary (192 processes) on /shared/home/vinil/run/motorBike
Running potentialFoam (192 processes) on /shared/home/vinil/run/motorBike
Running checkMesh (192 processes) on /shared/home/vinil/run/motorBike
Running simpleFoam (192 processes) on /shared/home/vinil/run/motorBike
vinil@ofcls-prod-scheduler:~/of2312-build-repo$

You can locate more detailed log files for each process OpenFOAM run within the case directory.

vinil@ofcls-prod-scheduler:~/of2312-build-repo$ ls -l $HOME/run/motorBike/log.*
-rw-rw-r-- 1 vinil vinil   2407 Feb 12 17:42 /shared/home/vinil/run/motorBike/log.blockMesh
-rw-rw-r-- 1 vinil vinil  10549 Feb 12 17:44 /shared/home/vinil/run/motorBike/log.checkMesh
-rw-rw-r-- 1 vinil vinil  95795 Feb 12 17:42 /shared/home/vinil/run/motorBike/log.decomposePar
-rw-rw-r-- 1 vinil vinil   2578 Feb 12 17:44 /shared/home/vinil/run/motorBike/log.patchSummary
-rw-rw-r-- 1 vinil vinil   3021 Feb 12 17:44 /shared/home/vinil/run/motorBike/log.potentialFoam
-rw-rw-r-- 1 vinil vinil   1572 Feb 12 17:47 /shared/home/vinil/run/motorBike/log.reconstructPar
-rw-rw-r-- 1 vinil vinil  52984 Feb 12 17:46 /shared/home/vinil/run/motorBike/log.reconstructParMesh
-rw-rw-r-- 1 vinil vinil 777220 Feb 12 17:46 /shared/home/vinil/run/motorBike/log.simpleFoam
-rw-rw-r-- 1 vinil vinil 276955 Feb 12 17:44 /shared/home/vinil/run/motorBike/log.snappyHexMesh
-rw-rw-r-- 1 vinil vinil   4156 Feb 12 17:42 /shared/home/vinil/run/motorBike/log.surfaceFeatureExtract
-rw-rw-r-- 1 vinil vinil   1801 Feb 12 17:44 /shared/home/vinil/run/motorBike/log.topoSet
vinil@ofcls-prod-scheduler:~/of2312-build-repo$

In conclusion, this blog has explored the seamless integration of OpenFOAM into high-performance computing environments facilitated by Azure CycleCloud. By configuring a Slurm-based HPC setup with HBv4 virtual machines and Ubuntu 22.04, engineers can optimize performance effectively. The post also delved into the crucial aspect of reducing OpenFOAM's compilation time and provided a comprehensive walkthrough for the compilation and execution process. Leveraging the parallel computing capabilities of MPI, along with Azure CycleCloud's scalable infrastructure, engineers can achieve remarkable simulation results. This demonstration underscores the immense potential of integrating OpenFOAM into HPC environments, heralding new horizons in fluid dynamics research and engineering applications.