Your H100s on Azure won't do distributed training. Here's probably why.

I recently had to dig into why a customer's H100 VMs couldn't do distributed training on Azure. They had ND-series capacity allocated, the hardware was there, but the VMs weren't connected over InfiniBand. The cause turned out to be a deployment configuration issue, not a hardware problem. Finding a single clear explanation of what went wrong and how to fix it was harder than it should have been.

The cause comes down to one of two things. Either they're on the wrong H100 SKU, or their VMs aren't in the same placement group. Both are fixable, but you have to understand what went wrong first.

The naming trap

Azure has two H100 VM families. The names look similar enough that people pick the wrong one without realizing it.

The NCads H100 v5 family has two sizes: NC40ads (1 GPU, 40 Gbps) and NC80adis (2 GPUs, 80 Gbps). Neither has InfiniBand. If you deploy 10 of these and try to run a multi-node training job, the GPUs communicate over standard Ethernet. For the all-reduce operations that distributed training depends on, you're looking at 40 Gbps per VM versus 3.2 Tbps aggregate IB on a single ND node, roughly 80x less bandwidth. In practice the gap is worse because Ethernet lacks GPUDirect RDMA, so every transfer bounces through main memory and the CPU.

The ND is the training machine. Eight H100 SXM GPUs per VM connected by NVLink internally, and each GPU gets its own dedicated 400 Gb/s InfiniBand link for talking to GPUs on other VMs. With GPUDirect RDMA, NCCL sends data directly between GPU memory across nodes, bypassing the CPU entirely.

If your training job needs more GPUs than fit in one VM, you need ND. No workaround for this.

When InfiniBand matters (and when it doesn't)

Not every GPU workload needs IB.

Single-GPU inference or dev work is fine on NC. No inter-node communication involved.

LoRA and QLoRA fine-tuning up to roughly 70B usually fits inside a single ND VM (8 GPUs with model parallelism). You're not crossing VM boundaries, so IB isn't in the picture.

Full-parameter SFT or RLHF on 8B+ is where it gets interesting. An 8B model in bf16 is about 16 GB just for the parameters. Full-parameter Adam training typically requires 12 to 16 bytes per parameter when you account for fp32 master weights, gradients, and optimizer first/second moments. That puts an unsharded 8B run at 96 to 128 GB before activations even enter the picture. With FSDP or ZeRO-3 sharding across the 8 GPUs inside a single ND VM, it fits without crossing node boundaries. But if you're trying to do this on standalone VMs without a sharding strategy, you'll need multi-node, and that means fast interconnect.

Large-scale pre-training at 70B+ runs across dozens or hundreds of GPUs. InfiniBand is not optional at this scale. Gradient synchronization over Ethernet would make the job impractical.

Rule of thumb: if your training job uses more GPUs than fit in one VM, InfiniBand starts mattering. More than two VMs, it matters a lot.

Making the VMs talk to each other

Having ND VMs in the same region isn't enough. They need to be on the same InfiniBand fabric to communicate at full speed.

Azure handles this through placement groups. A placement group is a set of VMs deployed on the same network fabric segment. VMs in the same group get non-blocking, full-bisection IB connectivity. VMs in different placement groups, or standalone VMs not in any VMSS or availability set, do not get InfiniBand communication with each other at all.

What happens when IB isn't available depends on your NCCL configuration. If NCCL_IB_DISABLE=1 is set, or NCCL can't find IB devices, it falls back to TCP over Ethernet (slow but functional). If IB interfaces exist but aren't connected to peers (the standalone ND case), NCCL may hang indefinitely waiting to establish the IB connection. Set NCCL_DEBUG=INFO in your environment to see which transport NCCL is actually using and where it gets stuck.

The following diagram illustrates this specific scenario: same ND hardware, deployed differently, completely different outcomes.

To get your NDs into a single placement group, deploy them in a Virtual Machine Scale Set (VMSS) with singlePlacementGroup set to true:

{ "type": "Microsoft.Compute/virtualMachineScaleSets", "properties": { "singlePlacementGroup": true } }

This works with both Flexible and Uniform orchestration modes (the IB setup docs confirm support for both). The key constraint is the placement group, not the orchestration mode. With singlePlacementGroup=true, the scale set is capped at 100 VMs by default (expandable to 300 via support request). For most startup training jobs, even the default cap gives you 800 H100 GPUs, which is plenty for post-training and medium-scale pre-training. If you need more than 2,400 GPUs on a single IB fabric, that's a different conversation involving Azure HPC team engagement.

The important thing: setting singlePlacementGroup=false (which enables large scale sets beyond 100 VMs) breaks InfiniBand networking entirely. The placement groups documentation states this explicitly: "Large scale (SPG=false) does not support InfiniBand networking." VMs land in different placement groups, and IB does not work across groups. If you need more than 100 nodes for IB training, open a support request to raise the single placement group limit rather than switching to multi-placement-group mode.

Once your NDs are in the same placement group, InfiniBand is automatically configured between VMs. No manual fabric setup required. The ND H100 v5 documentation confirms: "These connections are automatically configured between VMs occupying the same virtual machine scale set, and support GPU Direct RDMA."

Contiguous capacity has to be available in the region at deployment time. If the region is fragmented, the deployment might fail even though individual VMs would succeed. That's a capacity planning conversation, not a technical bug.

You can also use an availability set for the same purpose. VMs in the same availability set get IB connectivity. VMSS is usually the better option because it gives you scaling and lifecycle management, but availability sets work if you're managing VMs individually.

Docs: Set up InfiniBand on HPC VMs | Placement groups

You can't retrofit this without downtime

The natural follow-up question: "We already have ND VMs running as standalone instances. Can we rearrange them into a contiguous IB cluster without downtime?"

No. Azure doesn't offer a customer-facing operation to reposition VMs onto contiguous physical hardware. These H100 VM families do not support live migration, so there's no mechanism (customer-initiated or platform-initiated) that moves a running VM to a different physical host for network topology purposes.

The path is: deallocate your existing ND VMs, create a VMSS with singlePlacementGroup=true, and redeploy into the new scale set.

This means downtime. It also means contiguous capacity needs to be available in the region at that moment. If you're early in your deployment (and most startups are), set this up correctly from the start. Retrofitting later is possible but painful, especially if your region is capacity constrained.

The worst-case scenario here is real: you deallocate your VMs, try to redeploy into a placement group, and the region doesn't have enough contiguous rack space. Now you have no training cluster running and you're waiting for capacity to free up. If you're in a tight region, coordinate with your Microsoft account team to pre-validate contiguous availability before scheduling the maintenance window.

Images and drivers

Azure publishes HPC marketplace images (Ubuntu-HPC, AlmaLinux-HPC) that ship with NVIDIA GPU drivers, CUDA, NCCL, and Mellanox OFED pre-installed. Use these. They save you a day of driver debugging.

If you need a custom image or container, you'll need NVIDIA GPU drivers matching your CUDA version, Mellanox OFED for InfiniBand, and NCCL built with IB and GPUDirect RDMA support.

After deployment, validate the setup:

• ibstat should show IB devices in Active state on every VM. Here's what success looks like:
  
  

  CA 'mlx5_0' CA type: MT4129 Port 1: State: Active Physical state: LinkUp Rate: 400

  
  If you see State: Down or Physical state: Polling, the IB link isn't established. Check that your VMs are actually in the same placement group.
  
  
• lspci | grep Mellanox confirms the hardware is visible to the OS. You should see 8 ConnectX-7 devices on an ND96isr.
• Run NCCL all-reduce tests across nodes with NCCL_DEBUG=INFO set:
  
  

  export NCCL_DEBUG=INFO export NCCL_IB_DISABLE=0 mpirun -np 16 --hostfile hosts nccl-tests/build/all_reduce_perf -b 8 -e 8G -f 2 -g 1

With GPUDirect RDMA working correctly, you should see bus bandwidth approaching 380+ Gb/s for large messages (8 GB+). If you're seeing numbers well below that, or NCCL logs show NET/Socket instead of NET/IB, check that OFED is loaded (ofed_info), the IB interface is up (ibstat), and your VMs are in the same placement group.

Docs: Configure HPC VMs | ND H100 v5 specs

Before you start training

A quick checklist before launching your first distributed job:

• You're on ND96isr H100 v5, not NC40ads H100 v5
• VMs are in a VMSS with singlePlacementGroup=true (or same availability set). Standalone VMs won't have IB connectivity with each other.
• Using Ubuntu-HPC or AlmaLinux-HPC image (or custom with OFED + NCCL)
• ibstat shows IB Active on all nodes (all 8 ports per VM)
• NCCL all-reduce test confirms expected bandwidth for large messages
• NCCL logs show NET/IB transport, not NET/Socket
• Training framework configured with backend='nccl'

Further reading

1. ND H100 v5 specifications
2. NC H100 v5 specifications
3. Set up InfiniBand on HPC VMs
4. VMSS placement groups
5. Distributed GPU training guide (Azure ML)