What does it take to turn a brand-new GPU rack into a cloud service? See how Microsoft brings AMD Helios to Azure, from first power-on in the lab to burn-in at scale.
Article contributed by: Yash Bhamare and Anna Daly
Azure’s ND MI455X v7 brings a rack-scale approach to AI inference, powered by AMD Helios. Each liquid-cooled compute rack combines 72 MI455X GPUs, 18 AMD EPYC “Venice” CPUs, and 31 TB of HBM4 memory, with a two-tier Ethernet backend connecting racks into larger clusters. Designed for reasoning, long-context inference, search, and multi-agent services, the platform brings compute, memory, and networking together to support demanding AI workloads.
In the Azure context, “silicon to service” describes the engineering around that hardware foundation and deploying the hardware at scale.
Start with the user’s needs
Consider a user who asks a trained AI model a question and receives a response. For that user, the response is the result that matters. Microsoft provides the infrastructure behind that experience as a cluster product, including GPU capacity and a commitment to its availability and reliability. Users need confidence that their GPUs will be available when needed, that the next job will run reliably, that updates will leave the system usable, and that faults will leave enough evidence for diagnosis.
Reliability lets model teams and engineers focus on their models rather than rediscovering machine state. It requires infrastructure that can be deployed, observed, updated, and recovered predictably.
Engineering Behind the AI ResponseDatacenter requirements
At Microsoft, we deliver AI supercomputing clusters as a product. With AMD, we design from the rack up to meet customer workload requirements. We account for power, cooling, and physical fit so each rack can be deployed, operated, and maintained in the datacenter throughout its lifetime. A Microsoft-owned, two-tier back-end network connects racks for high-bandwidth communication.
Our job is to bring compute, networking, software, and physical infrastructure together into a system customers can use reliably. Hardware availability shapes how we get there, and a standalone rack lets us advance software enablement and rack-local testing, but validating the cluster requires connected racks and the back-end network operating together under load. That is why we include power delivery, cooling, cabling, and software versions in the test configuration rather than treating them as separate concerns.
The same principle applies after maintenance. We confirm that a replacement component responds and works within the connected system.
Microsoft’s tiered backend network
Establish a repeatable software foundation
System firmware, the operating-system kernel and device drivers must work together to initialize and manage hardware. Testing each layer alone cannot establish how the stack behaves.
A versioned baseline recipe of all firmware components, the operating-system image, GPU and network drivers, and diagnostic tools are tested together. This supports reproducible installation in cloud-scale datacenters and helps distinguish hardware faults from configuration or integration issues. Secure boot and driver signing also belong in that baseline and repeatability must preserve Microsoft’s trust requirements. When expected behavior changes, the baseline provides a traceable starting point for investigation.
Bring cloud management into the lab early
Our work starts when the first systems power on in the lab, and since early hardware is limited and still evolving, we focus on iterating quickly by testing whatever we can, fixing what we find, and testing again.
The first milestone is a stable foundation, where each system reliably boots the target operating system, discovers and initializes its devices, and reports their health. Getting there often exposes subtle interactions between firmware, drivers, and the operating system that we need to resolve before more advanced testing can begin.
Once the system is stable, we turn to performance, starting with microbenchmarks for GPU compute, memory bandwidth, and GPU-to-GPU and network transfers, and using tools such as TransferBench and RCCL tests to characterize how accelerators communicate. These simple tests are especially valuable during bring-up because an unexpected result can point to a firmware defect, a driver issue, or a configuration problem. From there, we step up to distributed training and inference workloads to see how compute, memory, and networking behave together under sustained load and where the software stack needs tuning.
Alongside this work, we validate how device inventory and configuration integrate with Azure for health reporting, updates, and recovery, so the tests, tooling, and management workflows we refine in the lab are ready to scale in the datacenter.
Burn-In – Make readiness an engineering outcome
Burn-in is where we take the same kinds of tests we ran in the lab and automate them at scale in the datacenter. Before a cluster reaches customers, we lock in an approved hardware and software configuration and use SuperBench to orchestrate sustained workloads across the whole system.
Running these workloads for extended periods across thousands of GPUs surfaces intermittent hardware failures, thermal problems, and performance anomalies that shorter lab runs can miss. The communication tests from the lab also expand to cover the full network, both within racks (scale-up) and across racks (scale-out), where RCCL collectives such as AllReduce and AllGather expose bandwidth bottlenecks, topology misconfigurations, and failures that only appear at scale.
After any faulty components are repaired or replaced and retested, the cluster enters production with data showing it can sustain demanding distributed AI workloads at expected performance and run reliably as one system.
Burn-In Process at Scale
Exercise recovery as normal operation
Once GPU racks enter production and begin running customer workloads, hardware failures are inevitable over time. These failures may manifest as workload performance degradation, unexpected errors, or device failures that must be traced to specific hardware components across one or more systems.
To maintain cluster capacity, healthy spare nodes replace affected systems, while faulty nodes are moved to dedicated quarantine racks for further investigation. There, we run extensive diagnostics to isolate the underlying issues and repair or replace faulty components.
Repair alone, however, is not sufficient to return a system to production. While still in quarantine, the repaired node undergoes another round of validation, running representative workloads against established performance and reliability baselines. This ensures that the original issue has been resolved and that the system performs as expected under realistic operating conditions.
Once the node passes these checks, it becomes a healthy spare, ready to replace the next system requiring maintenance. This creates a continuous cycle of detection, isolation, repair, validation, and reuse, allowing Azure to maintain reliable GPU capacity as hardware ages and individual components fail.
We apply the same rigor to system updates, looking across the GPU, firmware, operating system, and network layers to verify that the system remains healthy and workloads can continue or recover predictably after a change.
Come and See it at SC26
This is what we mean by Microsoft delivering AI Supercomputing Clusters. Yash Bhamare and Anna Daly will share more in their session at the Microsoft booth (#3411) at SC26, on Wednesday 18 November at 10:00am CT.
Microsoft