Forum Discussion
youbarajpaudel
Jun 08, 2023Copper Contributor
Setting up own HPC in Azure cloud
Hi All,
We recently moved many of our applications and databases to Azure Cloud. Some of the applications needs high computation power as these perform various types of mathematical simulations as part of insurance liability management.
We would like to set up and use Azure Cloud HPC instead of a third party tool. Based on the information on internet I got the impression that we can use Azure native services like Azure Batch or / and CycleCloud and perhaps other components to build a HPC. Does MS offer a ready HPC tool that scales up and down nodes as per client needs?
If not, any clear guidance on how we can set up such a tool?
For clarity, we want to use this tool for different applications, some of them are home-brew (C#) and others are vendor tools.
Many thanks.
- azharamir13Brass Contributor, Microsoft offers a High-Performance Computing (HPC) solution called Azure CycleCloud, which provides capabilities to scale up and down compute nodes based on client needs. Azure CycleCloud is a cloud-based HPC job and cluster management platform that simplifies the deployment, management, and scaling of HPC clusters in the Microsoft Azure cloud environment.
- youbarajpaudelCopper Contributor
Thanks for your response Amir! My understanding is that the cycle cloud is more meant for on-prem deployment while Azure batch could be used in cloud.
I cannot find clear guidelines about how to set up such a HPC in Azure and integrate it with different applications which need computing resources (nodes).
- azharamir13Brass ContributorMark as Like, if this helps
You are correct that Azure CycleCloud is often used for managing HPC clusters in on-premises or hybrid environments. While Azure Batch is a cloud-based service that provides job scheduling and resource management capabilities for HPC workloads.
To set up an HPC environment in Azure and integrate it with different applications, including those that require computing resources (nodes), you can follow these general guidelines:
Azure Batch Account: Start by creating an Azure Batch account. This account serves as the central management point for your HPC environment in Azure.
Create a Pool: Within your Azure Batch account, create a pool of compute resources. A pool consists of virtual machines (VMs) that will be used to execute your HPC workloads. You can choose the VM sizes and the number of VMs based on your specific requirements.
Job and Task Management: Define the jobs and tasks that need to be executed within your HPC environment. A job represents a collection of tasks that are scheduled for execution. Each task is a unit of work that will be processed on one of the compute nodes in your pool.
Application Integration: Configure your applications to integrate with Azure Batch. This typically involves packaging your application code and dependencies into containers or application packages that can be deployed to the compute nodes. Azure Batch supports various deployment options, including Docker containers, custom VM images, or using Azure Batch-specific APIs and SDKs.
Data Management: Ensure that your HPC environment has access to the necessary data and storage resources required by your applications. Azure offers a range of storage options, such as Azure Blob Storage or Azure Files, which can be used to store input data, intermediate results, and output files.