As agents generate more tokens and use longer context windows, AI platforms must reduce cost per token and optimize inference performance across every layer of the stack. With new native integrations between Azure Blob Storage and the NVIDIA Dynamo stack, users can now combine the scalability, durability, and operational simplicity of Azure Blob Storage with Dynamo’s high-performance inference capabilities.
NVIDIA Dynamo is a suite of tools enabling accelerated AI inference systems on Kubernetes. Dynamo enables disaggregated inference, intelligent Key-value (KV) cache-aware routing, optimized model weight management, high-performance data movement, and more through a collection of fully modular components.
Our measurements demonstrate that vLLM enhanced with Dynamo ModelExpress and Azure Blob Storage loads weights up to 7.3x faster than the default loader, translating into up to 4.6× faster total cold start latency. When leveraging NIXL’s integration with Azure Blob Storage for KV cache offloading, vLLM provides up to 2.8x lower time-to-first-token (TTFT) latency when compared to recomputing KV cache.
Accelerate Model Cold Starts with Dynamo ModelExpress
Modern AI models continue to grow in size, with frontier models regularly exceeding one trillion parameters. Each time an inference cluster scales up and launches a new model instance, the model’s weights must be loaded into GPU memory before serving can begin. These weights often exceed one terabyte in size, making weight-loading the single largest contributor to cold start latency.
Reducing this bottleneck is critical for adaptable inference. Faster cold starts allow clusters to scale more quickly, responding to traffic spikes with less delay, and reduce idle GPU time.
Dynamo ModelExpress addresses this challenge by optimizing the end-to-end lifecycle of model weights across a Dynamo inference cluster, coordinating how weights are loaded, cached, distributed, and reused. A core part of this optimization is its native integration with Run:ai Model Streamer, which accelerates cold-start loading by streaming model weights from Azure Blob Storage into GPU memory using optimized parallel transfer and prefetching. Run:ai Model Streamer leverages the full network bandwidth provided by the Azure VM’s NIC, maximizing transfer speeds from Blob storage. Together, Dynamo ModelExpress and Run:ai Model Streamer reduce model startup latency by accelerating the initial load and enabling subsequent reuse of model weights across the cluster.
In our testing across multiple node types on Azure, vLLM with Dynamo ModelExpress loaded weights from Azure Blob Storage into GPU memory up to 7.3x faster and began serving up to 4.6x faster compared with the default vLLM loader.
Figure 1 shows the performance boosts observed with ModelExpress versus the vLLM model loader across multiple models:
The results show that these optimizations become more impactful as model size increases. As model weight files grow larger, faster Blob-backed streaming has a larger effect on the full startup path. For more details about the hardware, configuration, and procedure used in these measurements, see the ModelExpress deployment recipe.
For deployments on Azure Kubernetes Service (AKS), storing model weights in Azure Blob Storage and loading them with Dynamo ModelExpress provides three major benefits:
- Faster Elastic Scaling: Model weights can be streamed from Azure Blob Storage into the Dynamo model loading path, reducing weight-loading time and cold-start latency.
- Operational simplicity: Teams can centrally store model weights in Azure Blob Storage and use Microsoft Entra ID and Azure RBAC to control access from across clusters.
- Enterprise-Grade Reliability: Azure Blob Storage provides lifecycle management and high durability for model artifacts, helping ensure that model weight files are securely stored and highly available for production inference workloads.
Extend KV Cache Offloading to Azure Blob
Cold start latency is only one challenge of the inference lifecycle. Once a model is deployed, a new challenge emerges: the KV data generated during inference is expensive to compute, so it must be efficiently cached and reused across requests to sustain high throughput and scalability.
When cached KV data is reused, inference clusters avoid recomputing the same data, helping requests achieve lower TTFT latency. While KV caching typically consumes GPU memory, offloading the KV cache to host memory or storage releases valuable GPU capacity while still enabling lower TTFT than recomputation.
KV cache management frameworks such as LMCache provide the platform that helps orchestrate this offloading. Many KV cache management tools leverage the NVIDIA Inference Xfer Library (NIXL) to facilitate the transfer of KV cache between GPU memory and host storage. NIXL is a core Dynamo component that provides high-bandwidth, low-latency point-to-point data transfers across various memory and storage tiers including HBM, DRAM, and local and remote storage. Through plugins, users can extend NIXL’s data transfers to specialized platforms such as Azure Blob Storage and GPUDirect Storage (GDS). Similar to when loading weights with ModelExpress, offloading KV cache to Blob storage is designed to efficiently use the network bandwidth available to the Azure VM.
NIXL’s plugin for Azure Blob Storage allows KV Cache management tools to offload KV cache directly to Azure Blob Storage. Offloading KV cache to Azure Blob Storage adds three concrete benefits for inference clusters on AKS:
- Reduced GPU Load: On a cache hit, the serving engine can reuse KV data from Azure Blob instead of recomputing the KV data.
- Scalable KV Caching Capacity: Blob’s scalability lets the cache layer keep more reusable KV blocks, preserving valuable GPU memory.
- Better KV Offload Effectiveness: More cached blocks can be retained and retrieved across requests, increasing the chance that a stored block is reused.
As shown below in Figure 2, offloading KV cache to Azure Blob storage yields up to a 2.8x improvement in TTFT versus without offloading. This significantly reduces user-facing latency and provides a much smoother experience.
See the NIXL Azure Blob Storage offloading recipe for details about the hardware, configuration, procedure, and other models used in our measurements.
Accelerate Your Inference Workloads Today
Azure Blob Storage’s integrations with Dynamo ModelExpress and NIXL provide multi-layered optimizations for accelerated inference on Azure Kubernetes Service (AKS), resulting in a streamlined inference lifecycle, including faster model readiness, reduced Time-to-First-Token (TTFT) latency, and more efficient GPU memory utilization.
Start optimizing your AI inference workloads today by leveraging the native integrations between Azure Blob Storage and the NVIDIA Dynamo stack. To see how to deploy these capabilities, explore the step-by-step guidance for Dynamo ModelExpress and Azure Blob Storage and the recipe for NIXL KV cache offloading to Azure Blob. Continue to follow NVIDIA and Microsoft as we deepen our integration to build the future of accelerated inference.
Acknowledgements
This work was enabled by close collaboration between engineering teams from Microsoft and NVIDIA. Key contributors include Vishnu Charan TJ and Kyle Knapp from Microsoft; and Stephen McCullough, Akshatha Kamath, Adit Ranadive, Nikhar Maheshwari, Ganesh Kudleppanavar, Vishwanath Venkatesan, and Zheng Luo from NVIDIA.
Microsoft