MPI job in cluster built with Standard D2s v3 (2 vcpus, 8 GB memory)

Hello everyone, I am new to Azure. And I tried to build a cluster with Ubuntu nodes in Azure. So I chose D2s v3 (2 vCPUs, 8 GB memory) and Ubuntu1604 Image to build tow VMs as the master node and worker node. I just followed the guidebook to create these two VMs and established /etc/hosts in each node and set up NFS so that the /home of the master node can be mounted on /home of the worker node. It worked, I can passwordless ssh to each node from the other.

I ping node001 from the master node:

 

miker2746@mt-test21:~/rm-test1$ 
miker2746@mt-test21:~/rm-test1$ ping node001
PING node001 (10.0.0.25) 56(84) bytes of data.
64 bytes from node001 (10.0.0.25): icmp_seq=1 ttl=64 time=0.963 ms
64 bytes from node001 (10.0.0.25): icmp_seq=2 ttl=64 time=0.553 ms
64 bytes from node001 (10.0.0.25): icmp_seq=3 ttl=64 time=0.622 ms
64 bytes from node001 (10.0.0.25): icmp_seq=4 ttl=64 time=0.748 ms
64 bytes from node001 (10.0.0.25): icmp_seq=5 ttl=64 time=0.709 ms
^C
--- node001 ping statistics ---
5 packets transmitted, 5 received, 0% packet loss, time 4057ms
rtt min/avg/max/mdev = 0.553/0.719/0.963/0.139 ms
miker2746@mt-test21:~/rm-test1$ 

 

 

But when I ran my MPI job in this cluster, it failed and showed some errors as follows:

 

miker2746@mt-test21:~/rm-test1$ mpiexec -n 2 -f ~/hostfile fds random_meshes.fds 
^C[mpiexec@mt-test21] Sending Ctrl-C to processes as requested
[mpiexec@mt-test21] Press Ctrl-C again to force abort
[mpiexec@mt-test21] HYDU_sock_write (../../utils/sock/sock.c:418): write error (Bad file descriptor)
[mpiexec@mt-test21] HYD_pmcd_pmiserv_send_signal (../../pm/pmiserv/pmiserv_cb.c:252): unable to write data to proxy
[mpiexec@mt-test21] ui_cmd_cb (../../pm/pmiserv/pmiserv_pmci.c:174): unable to send signal downstream
[mpiexec@mt-test21] HYDT_dmxu_poll_wait_for_event (../../tools/demux/demux_poll.c:76): callback returned error status
[mpiexec@mt-test21] HYD_pmci_wait_for_completion (../../pm/pmiserv/pmiserv_pmci.c:501): error waiting for event
[mpiexec@mt-test21] main (../../ui/mpich/mpiexec.c:1147): process manager error waiting for completion
miker2746@mt-test21:~/rm-test1$ 

I have built exactly the same cluster with similar VM size on AWS EC2 and Google Compute Engine before, they all worked very well and never showed these errors before.

 

 

I tried to find the answers and solutions to my problem by reading documents of Azure. And according to https://docs.microsoft.com/en-us/azure/virtual-machines/linux/sizes-hpc, it seems that only VM size of H16r, H16mr, A8 or A9 is allowed to run MPI jobs. Is that true? Or it just means that MPI jobs can experience low latency RDMA when they run in HPC cluster built with one of these 4 VM sizes.

 

So my questions are:

1. Can anyone tell me what's wrong with my cluster, why will I have this kind of errors. Is that because I haven't set up Virtual Network correctly?

2. Can I run MPI jobs on a cluster that built with VM sizes other than these 4?

3. If MPI jobs only allow running in these 4 VMs, how can I set up a cluster with this particular VM?

 

Thank you very much

 

Michael