Announcing the Public Preview of AMLFS 20: Azure Managed Lustre New SKU for Massive AI&HPC Workloads

Sachin Sheth - Principal PDM Manager

Brian Barbisch - Principal Group Software Engineering Manager

Matt White - Principal Group Software Engineering Manager

Brian Lepore - Principal Product Manager

Wolfgang De Salvador - Senior Product Manager

Ron Hogue - Senior Product Manager

Introduction

We are excited to announce the Public Preview of AMLFS Durable Premium 20 (AMLFS 20), a new SKU in Azure Managed Lustre designed to deliver unprecedented performance and scale for demanding AI and HPC workloads. 

Key Features 

• Massive Scale: Store up to 25 PiB of data in a single namespace, with up to 512 GB/s of total bandwidth. 

• Advanced Metadata Performance: Multi-MDS (Metadata Server) architecture dramatically improves metadata IOPS. In mdtest benchmarks, AMLFS 20 demonstrated more than 5x improvement in metadata operations. An additional MDS is provided for every 5 PiB of provisioned filesystem.   

• High File Capacity: Supports up to 20 billion inodes for maximum namespace size. 

Why AMLFS 20 Matters 

• Simplified Architecture: Previously, datasets larger than 12.5 PiB required multiple filesystems and complex management. AMLFS 20 enables a single, high-performance file system for massive AI and HPC workloads up to 25 PiB, streamlining deployment and administration. 

• Accelerated Data Preparation: The multi-MDT architecture significantly increases metadata IOPS, which is crucial during the data preparation stage of AI training, where rapid access to millions of files is required. 

• Faster Time-to-Value: Researchers and engineers benefit from easier management, reduced bottlenecks, and faster access to large datasets, accelerating innovation. 

Availability 

AMLFS 20 is available in Public Preview alongside the already existing AMLFS SKUs. For more details on other SKUs, visit the Azure Managed Lustre documentation. 

How to Join the Preview 

If you are working with large-scale AI or HPC workloads and would like early access to AMLFS 20, we invite you to fill out this form to tell us about your use case. Our team will follow up with onboarding details.