Optimal MPI Process Placement for Azure HB Series VMs

[jshelley]

MPI Process Pinning for HB-series VMs

For MPI applications, optimal pinning of processes can lead to significant application performance improvements for under subscribed systems. Before AMD introduced the Chiplet design a few years back, to get the optimal performance the user just needed to decide if their application performed better running all on the same socket or equally balanced across the sockets. However, with the introduction of the Chiplet design, it became more complicated. The following is a link to a diagram that may help to better understand the chiplet design

In the chiplet design, AMD has essentially integrated a bunch of smaller CPUs together to provide a socket with 64 cores (8 - 16 smaller CPUs with 4-8 cores each). To maximize the performance from each core it is important to balance the amount of L3 cache and memory bandwidth per core.  We will discuss how to do this below for the following Azure HB VM types using IntelMPI and OpenMPI/HPC-X.

Azure HB VM:

This instance comes with 60 AMD Naples cores. Each socket contains 8 numa domain with 4 cores each. One 4 core numa domain is held back for the hypervisor leaving 15 numa domains for the user. When undersubscribing the VM to get the desired resources/core it is desirable to equally balance the L3 cache and memory bandwidth between cores. To do this the user will need to select either 15, 30, 45, or 60 cores per node.

Metrics	Azure
	HB60rs	HB60rs	HB60rs	HB60rs
Cores (Physical)	15	30	45	60
RAM (GB)	224	224	224	224
Network (BW) (Gb/s)	100	100	100	100
Memory BW (GB/s)	250	250	250	250
RAM/Core (GB)	14.93	7.47	4.98	3.73
Network BW/Core (Gb/s)	6.67	3.33	2.22	1.67
Memory BW/Core (GB/s)	16.67	8.33	5.56	4.17

OpenMPI 4 / HPC-X:

Note: To print out the placement of the cores before the application is run add the flag --report-bindings

    --bind-to core --map-by ppr:1:numa (15 cores)

    --bind-to core --map-by ppr:2:numa (30 cores)

    --bind-to core --map-by ppr:3:numa (45 cores)

Intel MPI:

Note: To print out the placement of the cores before the application is run add the environment variable I_MPI_DEBUG=4

15 PPN:

-env I_MPI_PIN_PROCESSOR_LIST=$(echo "for (i=0;i<60;i+=4) for (j=0;j<1;j++) i+j" | bc | sed -z 's/\n/,/g;s/,$/\n/')

30 PPN:

-env I_MPI_PIN_PROCESSOR_LIST=$(echo "for (i=0;i<60;i+=4) for (j=0;j<2;j++) i+j" | bc | sed -z 's/\n/,/g;s/,$/\n/')

45 PPN:

-env I_MPI_PIN_PROCESSOR_LIST=$(echo "for (i=0;i<60;i+=4) for (j=0;j<3;j++) i+j" | bc | sed -z 's/\n/,/g;s/,$/\n/')

Azure HBv2 VM:

This instance comes with the 120 AMD Rome cores. Each socket contains 15 numa domain with 4 cores each. Two 4 core numa domain are held back for the hypervisor. When undersubscribing the HBv2 VM to get the desired resources/core it is desirable to equally balance the L3 cache and memory bandwidth between cores. To do this the user will need to select either 30, 60, 90, or 120 cores per node.

Metrics	Azure
	HB120rs_v2	HB120rs_v2	HB120rs_v2	HB120rs_v2
Cores (Physical)	30	60	90	120
RAM (GB)	448	448	448	448
Network (BW) (Gb/s)	200	200	200	200
Memory BW (GB/s)	345	345	345	345
RAM/Core (GB)	14.93	7.47	4.98	3.73
Network BW/Core (Gb/s)	6.67	3.33	2.22	1.67
Memory BW/Core (GB/s)	11.50	5.75	3.83	2.88

If you want to undersubscribe your VM to get the optimal about of resources per core for you application then you can pin your processes to get the optimal placement for the 30, 60, or 90 cores. To do this you will need to add the following environment variables to your MPI jobs.

OpenMPI 4 / HPC-X:

Note: To print out the placement of the cores before the application is run add the flag --report-bindings

    --bind-to core --map-by ppr:1:numa (30 cores)

    --bind-to core --map-by ppr:2:numa (60 cores)

    --bind-to core --map-by ppr:3:numa (90 cores)

Intel MPI:

Note: To print out the placement of the cores before the application is run add the environment variable I_MPI_DEBUG=4

30 PPN:

-env I_MPI_PIN_PROCESSOR_LIST=$(echo "for (i=0;i<120;i+=4) for (j=0;j<1;j++) i+j" | bc | sed -z 's/\n/,/g;s/,$/\n/')

60 PPN:

-env I_MPI_PIN_PROCESSOR_LIST=$(echo "for (i=0;i<120;i+=4) for (j=0;j<2;j++) i+j" | bc | sed -z 's/\n/,/g;s/,$/\n/')

90 PPN:

-env I_MPI_PIN_PROCESSOR_LIST=$(echo "for (i=0;i<120;i+=4) for (j=0;j<3;j++) i+j" | bc | sed -z 's/\n/,/g;s/,$/\n/')

Azure HBv3 VM:

This instance comes with the 120 AMD Milan cores. Each socket contains 2 numa domain with 30 cores each. 2 cores from 4 chiplets are held back for the hypervisor. When undersubscribing the HBv3 VM to get the desired resources/core it is desirable to equally balance the L3 cache and memory bandwidth between cores. To do this the user will need to select either 16, 32, 64, 96, or 120 cores per node. To simplify the optimal process placement for our customers, we have provided additional HBv3 VM sizes (HB120-16rs_v3, HB120-32rs_v3, HB120-64rs_v3, HB120-96rs_v3) than the standard HB120rs_v3 size. Below you can see a table of the resources per core when using the various sizes.

Metrics	Azure
	HB120-16rs_v3	HB120-32rs_v3	HB120-64rs_v3	HB120-96rs_v3	HB120rs_v3
Cores (Physical)	16	32	64	96	120
RAM (GB)	448	448	448	448	448
Network (BW) (Gb/s)	200	200	200	200	200
Memory BW (GB/s)	345	345	345	345	345
RAM/Core (GB)	28.00	14.00	7.00	4.67	3.73
Network BW/Core (Gb/s)	12.50	6.25	3.13	2.08	1.67
Memory BW/Core (GB/s)	21.56	10.78	5.39	3.59	2.88

If you are using the HBv120rs_v3 size and you want to undersubscribe your VM to get the optimal about of resources per core for you application then you can pin your processes to the same cores used by the 16, 32, 64, or 96 core VM sizes. To do this you will need to add the following environment variables to your MPI jobs.

OpenMPI 4 / HPC-X:

Note: To print out the placement of the cores before the application is run add the flag --report-bindings

16 PPN:

--bind-to cpulist:ordered --cpu-set 0,8,16,24,30,38,46,54,60,68,76,84,90,98,106,114

32 PPN:

--bind-to cpulist:ordered 

--cpu-set 0,1,8,9,16,17,24,25,30,31,38,39,46,47,54,55,60,61,68,69,76,77,84,85,90,91,98,99,106,107,114,115

64 PPN:

--bind-to cpulist:ordered

--cpu-set 0,1,2,3,8,9,10,11,16,17,18,19,24,25,26,27,30,31,32,33,38,39,40,41,46,47,48,49,54,55,56,57,60,61,62,63,68,69,70,71,76,77,78,79,84,85,86,87,90,91,92,93,98,99,100,101,106,107,108,109,114,115,116,117

96 PPN:

--bind-to cpulist:ordered

--cpu-set 0,1,2,3,4,5,8,9,10,11,12,13,16,17,18,19,20,21,24,25,26,27,28,29,30,31,32,33,34,35,38,39,40,41,42,43,46,47,48,49,50,51,54,55,56,57,58,59,60,61,62,63,64,65,68,69,70,71,72,75,76,77,78,79,80,81,84,85,86,87,88,89,90,91,92,93,94,95,98,99,100,101,102,103,106,107,108,109,110,111,114,115,116,117,118,119

Intel MPI:

Note: To print out the placement of the cores before the application is run add the environment variable I_MPI_DEBUG=4

16 PPN:

-genv I_MPI_PIN_PROCESSOR_LIST=0,8,16,24,30,38,46,54,60,68,76,84,90,98,106,114

32 PPN:

-genv I_MPI_PIN_PROCESSOR_LIST=0,1,8,9,16,17,24,25,30,31,38,39,46,47,54,55,60,61,68,69,76,77,84,85,90,91,98,99,106,107,114,115

64 PPN:

-genv I_MPI_PIN_PROCESSOR_LIST=0,1,2,3,8,9,10,11,16,17,18,19,24,25,26,27,30,31,32,33,38,39,40,41,46,47,48,49,54,55,56,57,60,61,62,63,68,69,70,71,76,77,78,79,84,85,86,87,90,91,92,93,98,99,100,101,106,107,108,109,114,115,116,117

96 PPN:

-genv I_MPI_PIN_PROCESSOR_LIST=0,1,2,3,4,5,8,9,10,11,12,13,16,17,18,19,20,21,24,25,26,27,28,29,30,31,32,33,34,35,38,39,40,41,42,43,46,47,48,49,50,51,54,55,56,57,58,59,60,61,62,63,64,65,68,69,70,71,72,75,76,77,78,79,80,81,84,85,86,87,88,89,90,91,92,93,94,95,98,99,100,101,102,103,106,107,108,109,110,111,114,115,116,117,118,119

Azure HBv4/HX VM:

These instance comes with the 176 AMD Genoa cores. Each socket contains 2 numa domain with 88 cores each. 2 cores from 8 chiplets are held back for the hypervisor. When undersubscribing the HBv4/HX VM to get the desired resources/core it is desirable to equally balance the L3 cache and memory bandwidth between cores. To do this the user will need to select either 24, 48, 96, 144, 176 cores per node. To simplify the optimal process placement for our customers, we have provided additional HBv4/HX VM sizes (HB176-24rs_v4/HX176-24rs, HB176-48rs_v4/HX176-48rs, HB120-96rs_v4/HX176-96rs, HB176-144rs_v4/HX176-144rs) than the standard HB176rs_v4/HX176-24rs size. Below you can see a table of the resources per core when using the various sizes.

Metrics	Azure
	HB176-24rs_v4/ HX176-24rs	HB176-48rs_v4/ HX176-48rs	HB176-96rs_v4/ HX176-96rs	HB176-144rs_v4/ HX176-144rs	HB176rs_v4/  HX176-176rs
Cores (Physical)	24	48	96	144	176
RAM (GB)	688/1408	688/1408	688/1408	688/1408	688/1408
Network (BW) (Gb/s)	400	400	400	400	400
Memory BW (GB/s)	800	800	800	800	800
RAM/Core (GB)	28.7/58.7	14.3/24.3	7.0/12.0	4.8/9.8	3.9/8.0
Network BW/Core (Gb/s)	16.7	8.4	4.2	2.8	2.3
Memory BW/Core (GB/s)	33.3	16.7	8.3	5.6	4.5

If you are using the HBv176rs_v4 or HX176rs size and you want to undersubscribe your VM to get the optimal about of resources per core for you application then you can pin your processes to the same cores used by the 24, 48, 96, or 144 core VM sizes. To do this you will need to add the following environment variables to your MPI jobs.

OpenMPI 4 / HPC-X:

Note: To print out the placement of the cores before the application is run add the flag --report-bindings

24 PPN:

--bind-to cpulist:ordered --cpu-set 0,6,12,20,28,36,44,50,56,64,72,80,88,94,100,108,116,124,132,138,144,152,160,168

48 PPN:

--bind-to cpulist:ordered 

--cpu-set   0,1,6,7,12,13,20,21,28,29,36,37,44,45,50,51,56,57,64,65,72,73,80,81,88,89,94,95,100,101,108,109,116,117,124,125,132,133,138,139,144,145,152,153,160,161,168,169

96 PPN:

--bind-to cpulist:ordered

--cpu-set    0,1,2,3,6,7,8,9,12,13,14,15,20,21,22,23,28,29,30,31,36,37,38,39,44,45,46,47,50,51,52,53,56,57,58,59,64,65,66,67,72,73,74,75,80,81,82,83,88,89,90,91,94,95,96,97,100,101,102,103,108,109,110,111,116,117,118,119,124,125,126,127,132,133,134,135,138,139,140,141,144,145,146,147,152,153,154,155,160,161,162,163,168,169,170,171

144 PPN:

--bind-to cpulist:ordered

--cpu-set 0,1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16,17,20,21,22,23,24,25,28,29,30,31,32,33,36,37,38,39,40,41,44,45,46,47,48,49,50,51,52,53,54,55,56,57,58,59,60,61,64,65,66,67,68,69,72,73,74,75,76,77,80,81,82,83,84,85,88,89,90,91,92,93,94,95,96,97,98,99,100,101,102,103,104,105,108,109,110,111,112,113,116,117,118,119,120,121,124,125,126,127,128,129,132,133,134,135,136,137,138,139,140,141,142,143,144,145,146,147,148,149,152,153,154,155,156,157,160,161,162,163,164,165,168,169,170,171,172,173

Intel MPI:

Note: To print out the placement of the cores before the application is run add the environment variable I_MPI_DEBUG=4

24 PPN:

-genv I_MPI_PIN_PROCESSOR_LIST=0,6,12,20,28,36,44,50,56,64,72,80,88,94,100,108,116,124,132,138,144,152,160,168

48 PPN:

-genv I_MPI_PIN_PROCESSOR_LIST=0,1,6,7,12,13,20,21,28,29,36,37,44,45,50,51,56,57,64,65,72,73,80,81,88,89,94,95,100,101,108,109,116,117,124,125,132,133,138,139,144,145,152,153,160,161,168,169

96 PPN:

-genv I_MPI_PIN_PROCESSOR_LIST=0,1,2,3,6,7,8,9,12,13,14,15,20,21,22,23,28,29,30,31,36,37,38,39,44,45,46,47,50,51,52,53,56,57,58,59,64,65,66,67,72,73,74,75,80,81,82,83,88,89,90,91,94,95,96,97,100,101,102,103,108,109,110,111,116,117,118,119,124,125,126,127,132,133,134,135,138,139,140,141,144,145,146,147,152,153,154,155,160,161,162,163,168,169,170,171

144 PPN:

-genv I_MPI_PIN_PROCESSOR_LIST=0,1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16,17,20,21,22,23,24,25,28,29,30,31,32,33,36,37,38,39,40,41,44,45,46,47,48,49,50,51,52,53,54,55,56,57,58,59,60,61,64,65,66,67,68,69,72,73,74,75,76,77,80,81,82,83,84,85,88,89,90,91,92,93,94,95,96,97,98,99,100,101,102,103,104,105,108,109,110,111,112,113,116,117,118,119,120,121,124,125,126,127,128,129,132,133,134,135,136,137,138,139,140,141,142,143,144,145,146,147,148,149,152,153,154,155,156,157,160,161,162,163,164,165,168,169,170,171,172,173

Pinning for Hybrid (MPI + OpenMP)

When running in hybrid mode on HBv3 you will need to exclude some cores to get the proper pinning. 

HB and HBv2

HB and HBv2 are laid out with chiplet represented as a numa domain. To get the optimal L3 cache usage you will only want to use 2, 3, or 4 threads per MPI rank. Below are the environment variables that you will need to set to get the optimal mpi placement. For HB you will want to only use 15 (2, 3, or 4 threads/rank) or 30 ( 2 threads/rank) mpi ranks. For HBv2 you will want to only use 30 (2, 3, or 4 threads/rank) or 60 ( 2 threads/rank) mpi ranks. 

OpenMPI 4 / HPC-X:

• --bind-to core
• --map-by ppr:<mpi ranks/numa>:numa:pe=<threads/mpi rank>

Example: If I wanted to run 30 MPI ranks on HBv2 and use 3 threads/rank (90 total cores) I would use the following options

• -np 30
• --bind-to core
• --map-by ppr:1:numa:pe=3
• OMP_NUM_THREADS=3

HBv3

• Under investigation. If you know of a clean way to do this with OpenMPI that is equivalent to what Intel MPI does please share in the comments.

Intel MPI:

• I_MPI_PIN=on
• I_MPI_PIN_DOMAIN cache3
• OMP_NUM_THREADS=[2,3, or 4]

Example: If I wanted to run 30 MPI ranks on HBv2 and use 2 threads/rank (60 total cores) I would use the following options

• -np 30 (or some multiple of 30 * number of VMs)
• I_MPI_PIN=on
• I_MPI_PIN_DOMAIN cache3
• OMP_NUM_THREADS=2

HBv3

The approach that we found that works with Intel MPI is to exclude the cores you do not want it to use and then by using the I_MPI_PIN_DOMAIN variable you can get it properly use the remaining cores. Below is the list of cores you would want to exclude if you were to run 96 (exclude 24), 64 (exclude 56), or 32 (exclude 88) cores/node.

Exclude Cores
Cores	Core List
24	14,15,22,23,6,7,44,45,52,53,36,37,74,75,82,83,66,67,104,105,112,113,96,97
56	4,5,12,13,14,15,20,21,22,23,28,29,6,7,34,35,42,43,44,45,50,51,52,53,58,59,36,37,64,65,72,73,74,75,80,81,82,83,88,89,66,67,94,95,102,103,104,105,110,111,112,113,118,119,96,97
88	2,3,4,5,10,11,12,13,14,15,18,19,20,21,22,23,26,27,28,29,6,7,32,33,34,35,40,41,42,43,44,45,48,49,50,51,52,53,56,57,58,59,36,37,62,63,64,65,70,71,72,73,74,75,78,79,80,81,82,83,86,87,88,89,66,67,92,93,94,95,100,101,102,103,104,105,108,109,110,111,112,113,116,117,118,119,96,97

Recommendations for the following hybrid scenarios:

• Note: If you use other combinations of ranks and threads you will not have optimal resource distribution for L3 cache and will span AMD chiplets which will reduce performance

MPI Ranks	Threads/MPI rank	Exclude Cores
16	6	24
32	3	24
48	2	24
16	4	56
32	2	56
16	2	88

To run in hybrid mode, you will want to set the following environment variables

• I_MPI_PIN=on
• I_MPI_PIN_DOMAIN <threads/mpi rank>:compact
• I_MPI_PIN_PROCESSOR_EXCLUDE_LIST=<exclude core list>

Example: For 16 MPI ranks/node with 6 threads/rank ( 96 cores/node):       

• I_MPI_PIN=on
• I_MPI_PIN_DOMAIN 6:compact
• I_MPI_PIN_PROCESSOR_EXCLUDE_LIST=14,15,22,23,6,7,44,45,52,53,36,37,74,75,82,83,66,67,104,105,112,113,96,97

#AzureHPC #AzureHPCAI