Blog Post

Azure High Performance Computing (HPC) Blog
12 MIN READ

Performance at Scale: The Role of Interconnects in Azure HPC & AI Infrastructure

HugoAffaticati's avatar
HugoAffaticati
Icon for Microsoft rankMicrosoft
Jun 25, 2025

Microsoft Azure’s high-performance computing (HPC) & AI infrastructure is designed from the ground up to support the world’s most demanding workloads. High-performance AI workloads are bandwidth-hungry and latency-sensitive. As models scale in size and complexity, the efficiency of the interconnect fabric—how CPUs, GPUs, and storage communicate—becomes a critical factor in overall system performance. Even with the fastest GPUs, poor interconnect design can lead to bottlenecks, underutilized hardware, and extended time-to-results. In this blog post, we will highlight one of the key enabling features for running large-scale distributed workloads on Azure: a highly tuned HPC-class interconnect. Azure has invested years of system-level engineering of the InfiniBand interconnect, into ready-to-use configurations for customers available on Azure’s HB-series and N-series virtual machine (VMs).

by Hugo Affaticati (Cloud Infrastructure Engineer), Amirreza Rastegari (Senior Software Engineer), Jie Zhang (Principal Software Engineer), and Michael Ringenburg (Principal Software Engineer Manager...
Updated Jun 25, 2025
Version 4.0
ai infrastructure
benchmarking
hpc
virtual machines
joshua_mora's avatar
joshua_mora
Copper Contributor
Jul 02, 2025

Could you please post the actual network bandwidth utilization of those workloads at scale ? Network microbenchmarks are useful to validate scale up and scale out and are a marketing tool to justify the high bandwidth between accelerators. However real distributed and scalable (strong scaling) applications have way lower network bandwidth requirements, hence an opportunity to lean on networking infrastructure at large scale. 1/8th of the max bandwidth is sufficient for scale out. That is only 1x NIC @ 400G for every 8 GPUs is sufficient at very large scale. Maybe 2, one per group of CPU+4GPUs.

Also could you comment on reliability challenges observed at large scale (8k GPUs) where Mean Time Between Failures is a few hours? What processes are in place to recover fast on  distributed training at large scale for a 1month training of a foundational model that is disrupted every few hours due to unavoidable hw/sw failures ?

Reducing number of network ports not only is cheaper but proportionally more network reliable (less NICs, transceivers, cables, switches) without compromising the scalability of the applications.

Finally, please post your training throughput numbers with and without sharp at scale so it can be quantified SHARP value if any.