Azure VMs deliver inference performance in line with industry benchmarks

Article contributed by: Dilip Patlolla, Wenqing Lan, Mingfeng Yang, Ali Vaziri, Aishwarya Prasad Tonpe, Huseyin Yildiz, Anshul Jain and Aditya Bhamidipati

Azure delivers AI inference performance on par with industry benchmarks. We evaluated Azure ND GB200 v6 and ND GB300 v6 virtual machines across multiple models and inference engines, comparing ND GB200 v6 VMs with matched bare-metal servers and ND GB300 v6 VMs with the published InferenceX reference. Across the tested configurations, Azure closely matched these baselines in throughput and latency, with selected ND GB200 v6 workloads showing gains over bare metal. These results demonstrate Azure’s ability to deliver responsive, high-throughput inference at performance levels consistent with industry reference systems. 

We evaluated multiple dimensions of inference performance. Time to first token (TTFT) is the time between sending an LLM inference request and receiving the first generated output token. Time per output token (TPOT), sometimes called decode speed, is how quickly the rest of the answer streams. End-to-end latency is the total wait for the full response. Throughput is the total output a GPU produces per second, across all users.  

ND GB200 v6: Parity with bare metal, with selected gains 

Azure VMs show higher average throughput and faster first tokens. Decode and end-to-end response times remain effectively unchanged; gains vary by workload. 

Figure 1. ND GB200 v6 VM versus bare metal, four GPUs per node; 264 matched comparisons. Models/datatypes: DeepSeek-R1-0528 (FP4/FP8), GPT-OSS-120B (FP4), and Kimi-K2.5 (INT4). ISL/OSL (input/output tokens): 1024/1024, 1024/8192, and 8192/1024. Filled points show Azure; open points mark bare metal. Values are report averages on a zoomed axis, not confidence intervals.

These workloads test different serving pressures: longer prompts emphasize input processing, while longer answers emphasize sustained decoding. Looking across balanced, input-heavy, and output-heavy requests makes the comparison more informative than a single peak-throughput result. The VM and bare-metal environments still differ, so individual gains should not be attributed to virtualization alone. 

ND GB300 v6: Azure VMs closely match the InferenceX reference 

Across matched configurations, throughput and latency closely track the reference. The chart summarizes average relative differences for each metric. These runs use disaggregated serving: prefill GPUs process the prompt, while decode GPUs generate the response. Matching both stages matters because the allocation between them affects first-token timing, sustained generation, and aggregate throughput. 

Figure 2. ND GB300 v6 Azure versus InferenceX; 91 matched points for DeepSeek-R1-0528 (FP4/FP8), with ISL/OSL (input/output tokens) of 1024/1024 and 8192/1024. Open points mark zero difference; filled points show Azure mean relative differences on a -3% to +3% axis. Orange highlights TTFT; gray denotes changes below 0.1%. Mean/median metric names refer to request-level statistics; P99 metrics are omitted.

What we found on ND GB300 v6 

• Throughput: Essentially identical across the tested concurrency sweep. This is agreement across matched operating points, not just at a single peak-throughput setting.  

• Decode: Time per output token (TPOT), inter-token latency (ITL), and interactivity closely match. Once generation starts, Azure sustains a comparable streaming rate, with no meaningful steady-state decode penalty observed. 

• End-to-end latency: User-perceived response times remain operationally equivalent. This checks the complete request, not just first-token timing or the rate at which later tokens arrive. 

• First token: Typical time to first token (TTFT) matches. Some low-concurrency cases show wider variation, potentially reflecting first-request warm-up. These cases should be read alongside decode and end-to-end behavior, which remain closely aligned. 

• Precision and serving stack: FP4, FP8, Dynamo+SGLang, and Dynamo+TRT-LLM match, including tested multi-token prediction (MTP) variants. In plain terms, the result holds across the number formats (precisions) and serving software we tested. 

Matching throughput and responsiveness 

The overlay asks a practical question: can Azure deliver the same aggregate throughput at a comparable per-user generation rate? Follow each color as concurrency changes. Closely aligned curves indicate a similar operating envelope; different colors represent different configurations, not a single universal scaling curve. 

Figure 3. DeepSeek-R1-0528 on ND GB300 v6: FP4/FP8, with ISL/OSL (input/output tokens) of 1024/1024 and 8192/1024. Azure (solid) and InferenceX (dashed) curves for six densely sampled configurations. Matching colors identify configurations; both axes are logarithmic. Interactivity = tokens/s/user; throughput = total tokens/s/GPU; P/D = prefill/decode GPUs.

What this means for your inference workloads 

These results reflect how we build Azure, the same whole-system engineering behind our largest AI supercomputers goes into the virtual machines every Azure customer uses.  

The key takeaway is straightforward - Azure VMs can deliver industry-reference inference performance without an automatic VM performance discount. Across the tested workloads, Azure ND GB200 v6 VMs perform in line with bare-metal servers overall, while Azure ND GB300 v6 VMs closely track the published InferenceX reference. Together, these results support Azure VMs as a platform for responsive, high-throughput inference across the models, serving stacks, and scales evaluated. 

Come and See it at SC26 

GPU Inference at Parity, with Wenqing Lan and Mingfeng Yang, on Wednesday 18 November at 3:30pm CT at the Microsoft booth (#3411), Exhibit Hall. We'll walk through how we tested and what these results mean for large-model inference.