Setting Up Slurm Cloud Bursting Using CycleCloud on Azure

Azure CycleCloud is an enterprise-friendly tool for orchestrating and managing High-Performance Computing (HPC) environments on Azure. With CycleCloud, users can provision infrastructure for HPC systems, deploy familiar HPC schedulers, and automatically scale the infrastructure to run jobs efficiently at any scale.

Slurm is a widely used open-source HPC scheduler that can manage workloads across clusters of compute nodes. Slurm can also be configured to interact with cloud resources, such as Azure CycleCloud, to dynamically add or remove nodes based on the demand of the jobs. This allows users to optimize their resource utilization and cost efficiency, as well as to access the scalability and flexibility of the cloud.

In this blog post, we are discussing how to integrate an external Slurm Scheduler to send jobs to CycleCloud for cloud bursting (Enabling on-premises workloads to be sent to the cloud for processing, known as “cloud bursting”) or hybrid HPC scenarios. For demonstration purposes, we are creating a Slurm Scheduler node in Azure as an external Slurm Scheduler in a different VNET and the execute nodes are in CycleCloud in a separate VNET. We are not discussing the complexities of networking involved in Hybrid scenarios.

Prerequisites

Before proceeding, ensure you have the following requirements in place:  

• An Azure subscription

• Operating System: AlmaLinux HPC 8.x, Ubuntu HPC 20.04, and Ubuntu HPC 22.04 (Supported platform images in CycleCloud)  

• CycleCloud Version: 8.x  

• CycleCloud-Slurm Project: 3.0.x  

• CycleCloud Server: Must be up and running, and accessible via the `cyclecloud` CLI.

•  

  An external Slurm Scheduler node in Azure or on-premises. in this example we are using Azure VM running with Alma Linux 8.7.
• A network connection between the external Slurm Scheduler node and the CycleCloud cluster. You can use Azure Virtual Network peering, VPN gateway, ExpressRoute, or other methods to establish the connection. In this example, we are using a very basic network setup.
• A shared file system between the external Slurm Scheduler node and the CycleCloud cluster. You can use Azure NetApp Files, Azure Files, NFS, or other methods to mount the same file system on both sides. In this example, we are using a Scheduler VM as a NFS server.

Steps 

After we have the prerequisites ready, we can follow these steps to integrate the external Slurm Scheduler node with the CycleCloud cluster:

1. On CycleCloud VM:

• Ensure CycleCloud VM is running and accessible via cyclecloud CLI. 
• Clone this repository and run the sh 01_prep_cyclecloud-slurm_template.sh script in the cyclecloud directory.
• sh 01_prep_cyclecloud-slurm_template.sh give the CycleCloud project version, Slurm version and cyclecloud-slurm template to build the Cloud busting setup.

git clone https://github.com/vinil-v/slurm-cloud-bursting-using-cyclecloud.git
cd slurm-cloud-bursting-using-cyclecloud/cyclecloud/
sh 01_prep_cyclecloud-slurm_template.sh

Output:

[vinil@ccvm cyclecloud]$ sh 01_prep_cyclecloud-slurm_template.sh 
Project version: 3.0.10
Slurm version:  24.05.4-2
Template location : slurm-3.0.10/templates/slurm-headless.txt
Please refer README for customizing the template for Headless Slurm cluster
[vinil@ccvm cyclecloud]$ 

This above output shows the Cyclecloud-Slurm Project version available in your cyclecloud environment, supported slurm version in the project and the template for creating headless slurm cluster.

• Edit the template from the given template location (`Template location: slurm-3.0.10/templates/slurm-headless.txt`) using your favorite editor and make the following adjustment to create a headless template.
• Ensure that the `slurm` and `munge` UID and GID are included in the template under the `[[node defaults]]` and `[[[configuration]]]` sections if they are not already present. This ensures consistency with the scheduler's UID and GID for `munge` and `slurm`.

[[node defaults]]     
    [[[configuration]]]
    ....
        #slum and munge users and setting their ids
        slurm.user.name = slurm
        slurm.user.uid = 11100
        slurm.user.gid = 11100
        munge.user.name = munge
        munge.user.uid = 11101
        munge.user.gid = 11101

•  Remove the following sections completely in the template to prepare a headless template.

        [[node scheduler]]
        [[nodearray scheduler-ha]]
        [[nodearray login]]
        [[[parameter SchedulerMachineType]]]
        [[[parameter loginMachineType]]]
        [[[parameter NumberLoginNodes]]]
        [[[parameter SchedulerHostName]]]
        [[[parameter SchedulerImageName]]]
        [[[parameter LoginImageName]]]
        [[[parameter SchedulerClusterInitSpecs]]]
        [[[parameter LoginClusterInitSpecs]]]
        [[[parameter SchedulerZone]]]
        [[[parameter SchedulerHAZone]]]

• Once the headless template is prepared then run sh 02_cyclecloud_build_cluster.sh script to import the headless cluster to cyclecloud.
• in this example we use hpc10 as the cluster name.

sh 02_cyclecloud_build_cluster.sh

Output:

[vinil@ccvm cyclecloud]$ sh 02_cyclecloud_build_cluster.sh 
Enter Cluster Name: hpc10
Cluster Name: hpc10
Importing Cluster
Importing cluster Slurm_HL and creating cluster hpc10....
-----------
hpc10 : off
-----------
Resource group: 
Cluster nodes:
Total nodes: 0
Cluster Name: hpc10
Project version: 3.0.10
Slurm version:  24.05.4-2
[vinil@ccvm cyclecloud]$ 

Please make a note of the Cluster Name, Project version and Slurm version. which you will be using the next steps.

2. Preparing Scheduler VM:

•  Deploy a VM using the supported AlmaLinux HPC or Ubuntu HPC image.
• Run the Slurm scheduler installation script (`sh 01_slurm-scheduler-builder.sh`) and provide the `cluster name` and the `slurm version` when prompted.
•  Cluster Name and Slurm version should be same taken from the previous step.
•  This script will install and configure Slurm Scheduler.

git clone https://github.com/vinil-v/slurm-cloud-bursting-using-cyclecloud.git
cd  slurm-cloud-bursting-using-cyclecloud/scheduler/
sh 01_slurm-scheduler-builder.sh

Output:

[root@scheduler scheduler]# sh 01_slurm-scheduler-builder.sh 
------------------------------------------------------------------------------------------------------------------------------
Building Slurm scheduler for cloud bursting with Azure CycleCloud
------------------------------------------------------------------------------------------------------------------------------
 
Enter Cluster Name: hpc10
Enter the Slurm version to install (You get this from the cyclecloud_build_cluster.sh): 24.05.4-2                                            
------------------------------------------------------------------------------------------------------------------------------
 
Summary of entered details:
--------------------------
Cluster Name: hpc10
Scheduler Hostname: scheduler
NFS Server IP Address: XX.XXX.X.X
 
 Please Note down the above details for configuring cyclecloud UI
------------------------------------------------------------------------------------------------------------------------------
------------------------------------------------------------------------------------------------------------------------------
Creating Munge and Slurm users
------------------------------------------------------------------------------------------------------------------------------
Munge and Slurm users created
------------------------------------------------------------------------------------------------------------------------------
Setting up NFS server
------------------------------------------------------------------------------------------------------------------------------
<--- truncated output --->
------------------------------------------------------------------------------------------------------------------------------
Slurm installed
------------------------------------------------------------------------------------------------------------------------------
------------------------------------------------------------------------------------------------------------------------------
Configuring Slurm
------------------------------------------------------------------------------------------------------------------------------
 
------------------------------------------------------------------------------------------------------------------------------
Slurm configured
------------------------------------------------------------------------------------------------------------------------------
 
------------------------------------------------------------------------------------------------------------------------------
 Go to CycleCloud Portal and edit the hpc10 cluster configuration to use the external scheduler and start the cluster.
 Use XX.XXX.X.X IP Address for File-system Mount for /sched and /shared in Network Attached Storage section in CycleCloud GUI 
 Once the cluster is started, proceed to run  cyclecloud-integrator.sh script to complete the integration with CycleCloud.
------------------------------------------------------------------------------------------------------------------------------
 

3. CycleCloud UI:

• Access the CycleCloud UI, edit the `hpc10` cluster settings, and Select VM SKUs as per your requirement and select the `Subnet` Networking settings.
• Select the Appropriate OS Images for HPC or HTC nodearray (Select the exact same OS version of the Scheduler VM) under `Advanced Settings`.
• disable the `Return Proxy`.
• Enter the NFS server IP address for `/sched` and `/shared` mounts in the Network Attached Storage section.
• Save & start `hpc10` cluster

4. On Slurm Scheduler Node:

•  Integrate Slurm scheduler with CycleCloud using the `sh 02_cyclecloud-integrator.sh` script.
•  Provide CycleCloud details (username, password, and URL) when prompted.

cd  slurm-cloud-bursting-using-cyclecloud/scheduler/
sh sh 02_cyclecloud-integrator.sh

Output:

[root@scheduler scheduler]# sh 02_cyclecloud-integrator.sh 
Please enter the CycleCloud details to integrate with the Slurm scheduler
 
Enter Cluster Name: hpc10
Enter CycleCloud Username: vinil
Enter CycleCloud Password: 
Enter the Project version: 3.0.10
Enter CycleCloud URL (e.g., https://10.0.0.1): https://xx.xxx.x.x
------------------------------------------------------------------------------------------------------------------------------
 
Summary of entered details:
Cluster Name: hpc10
CycleCloud Username: vinil
CycleCloud URL: https://xx.xxx.x.x
 
------------------------------------------------------------------------------------------------------------------------------
------------------------------------------------------------------------------------------------------------------------------
Configuring virtual enviornment and Activating Python virtual environment
------------------------------------------------------------------------------------------------------------------------------
------------------------------------------------------------------------------------------------------------------------------

------------------------------------------------------------------------------------------------------------------------------

5. User and Group Setup:

• Ensure consistent user and group IDs across all nodes.
• Better to use a centralized User Management system like LDAP to ensure the UID and GID are consistent across all the nodes.
• In this example we are using the `useradd_example.sh` script to create a test user `user1` and group for job submission. (User `user1` is exist in CycleCloud)

cd slurm-cloud-bursting-using-cyclecloud/scheduler
sh useradd_example.sh

Output:

[root@scheduler scheduler]# sh useradd_example.sh 
Enter User Name: user1
Generating public/private rsa key pair.
Your identification has been saved in /shared/home/user1/.ssh/id_rsa.
Your public key has been saved in /shared/home/user1/.ssh/id_rsa.pub.
The key fingerprint is:
SHA256:dURAjAhAQqTHNp3VnhyaeRyVk36OirzfatWk/8+onLw user1@scheduler
The key's randomart image is:
+---[RSA 3072]----+
|++o....o =+=o    |
|.o . o. = =.     |
|. = o  B =...    |
| o .  + *...o    |
|       .S  B     |
|          + o    |
|     . . o .     |
|      o o. o.. o |
|      .+o.. E+o.o|
+----[SHA256]-----+
[root@scheduler scheduler]# 

6. Testing & Job Submission:

• Log in as a test user (`user1` in this example) on the Scheduler node.
• Submit a test job to verify the setup.

su - user1
srun hostname &

Output:

[root@scheduler scheduler]# su - user1
Last login: Tue Feb 11 08:55:50 UTC 2025 on pts/0
[vinil@scheduler ~]$ srun hostname &
[1] 47480
[vinil@scheduler ~]$ squeue 
             JOBID PARTITION     NAME     USER ST       TIME  NODES NODELIST(REASON)
                 1       hpc hostname    user1 CF       0:03      1 hpc10-hpc-1
[vinil@scheduler ~]$ 

You should see the job running successfully, indicating a successful integration with CycleCloud.

For further details and advanced configurations, refer to the scripts and documentation within this repository.

These instructions provide a comprehensive guide for setting up Slurm bursting with CycleCloud on Azure. If you encounter any issues or have questions, please refer to the provided scripts and documentation for troubleshooting steps. Happy bursting!

NOTE: Lockdown environment need additional changes in the way we use the project and configure it. These are tested in the non-lockdown scenarios.

Congratulations! You have successfully set up Slurm bursting with CycleCloud on Azure.

Conclusion

In this blog post, we have shown how to integrate an external Slurm Scheduler node with Azure CycleCloud for cloud bursting or hybrid HPC scenarios. This enables users to leverage the power and flexibility of the cloud for their HPC workloads, while maintaining their existing Slurm workflows and tools. We hope this guide helps you to get started with your HPC journey on Azure.

References:

• GitHub repo - slurm-cloud-bursting-using-cyclecloud 
• Azure CycleCloud Documentation 
• Slurm documentation 
• cyclecloud-slurm