Azure HPC Cache Updates: New Caching Option, Discounted Pricing, and More!
New Releases Azure HPC Cache Premium Read-Write We’re excited to announce the preview of Azure HPC Cache Premium Read-Write. This next generation of premium caching for high-performance computing workloads is designed to provide high-bandwidth and low-latency access to files. Azure compute clients are provided with read and write performance like what they would experience from a local NVMe drive. Premium HPC Cache provides lower latency than the Standard HPC Cache for your compute-intensive enterprise workloads. You can provision up to 84 TB of capacity in a single cache and point thousands of compute clients at the cache to get up to 20 GB/s of read throughput. Premium HPC Cache’s highlights include: Increased Read Throughput: up to 20 GB/sec against an 80+ TiB dataset Reduced Latency: 150 µsec for reads, 1 msec for writes Increased Write Throughput: 24.5% increase in write operations IOPS Scalability: 170,000 random 4 KiB writes; 450,000 random 4 KiB reads With our lowest-latency cache, Premium HPC Cache provides our best file-based performance to meet your time-sensitive workloads like media rendering, simulations for genomics and financial models, as well as chip design. Getting Started with Premium Read-Write Azure HPC Cache Premium Read-Write is currently available as a Public Preview in select regions. If you are interested in participating in the preview, you can create a new cache and choose the Premium (Preview) option. Overview of Current and New Offerings Attribute Read-Write Standard Caching Read-Only Caching Read-Write Premium Caching Throughput SKU 2, 4, or 8 GB/sec 4.5, 9, or 16 GB/sec 5, 10, or 20 GB/sec Write Throughput 1.1, 2.2, 4.4 GB/sec N/A GB/sec (write-through) 2.3, 4.6, 9.2 GB/sec Read IOPS 62.5K, 125K, 250K ops/sec 160K, 320K, 480K ops/sec 500K, 1M, 2M ops/sec Write IOPS 16.75K, 33.5K, 67K ops/sec N/A ops/sec (write-through) 135K, 270K, 540K ops/sec Cache sizes 3, 6, or 12 TB for 2 GB/sec 6, 12, or 24 TB for 4 GB/sec 12, 24, or 48 TB for 8 GB/sec 21 TB for 4.5 GB/sec 42 TB for 9 GB/sec 84 TB for 16 GB/sec 21 TB for 5 GB/sec 42 TB for 10 GB/sec 84 TB for 20 GB/sec Maximum number of storage targets 20 20 20 Compatible storage target types Azure Blob, NFS (on-premises), ADLS-NFS (NFSv3-enabled Azure Blob) NFS (on-premises), ADLS-NFS (NFSv3-enabled Azure Blob) Azure Blob, NFS (on-premises), ADLS-NFS (NFSv3-enabled Azure Blob) Caching styles Read caching or read-write caching Read caching only Read-write caching Cache can be stopped to save cost when not needed Yes No No *Results were without use of the priming feature. Additional details on priming will be included in our next blog post. Lower Pricing for Azure HPC Cache - Standard While it seems like the cost of everything is going up, we’re happy to report a price drop. As part of our commitment to provide the most cost-effective performance cache, we’re excited to share that we have dropped Azure HPC Cache – Standard prices by up to 33% percent in some regions. The new pricing is effective immediately. This enables cost-conscious verticals like media and entertainment to meet strict rendering deadlines while spending less on caching. Life sciences customers who rely on grants to fund their genomics research can now stretch their funds further. And chip designers can run their EDA workloads at a lower cost and still maintain the high performance for their tools repositories that they’ve come to expect. Terraform Terraform, an open-source software tool created by HashiCorp, provides an orchestration layer to deploy Azure resources. Using Terraform, you can deploy your own vdbench test system with all the required resources to run a performance benchmark. To try this yourself, the HPC Cache team has created a vdbench NFS-backed storage cache Terraform recipe to deploy a Premium 5G cluster and 30 clients. You can additionally run your own benchmarks against a 5G storage cache. Instructions and examples can be found on HPC Cache GitHub.
📢 [Public Preview] Accelerating BlobNFS throughput & scale with FUSE for superior performance
Azure Blob Storage can be mounted and accessed like a local file system using BlobFuse, which is a FUSE-based driver for the Blob REST API. Customers choose BlobFuse for AI/ML, HPC, analytics and backup workloads. It provides exceptionally high throughput along with benefits like local caching and security integration via Azure Entra ID. For customers requiring NFS 3.0 protocol support or POSIX compliance, Azure Blob Storage also natively supports NFSv3 (aka BlobNFS). It enables Azure Blob storage access for customers’ legacy applications without requiring changes. BlobNFS is accessed via the Linux NFS client combined with our AZNFS mount helper package, which streamlines mounting and reliably connecting to Blob Storage’s NFS endpoints. Please note that BlobNFS only supports access over a virtual network since Azure Entra ID based auth isn’t yet available on NFS 3.0. Today, we’re excited to announce an update to AZNFS (3.0) for BlobNFS, which now uses the same libfuse3 library that powers BlobFuse bringing significant improvements in performance and scale. The updated AZNFS for BlobNFS delivers significantly higher throughput, larger file support, better metadata performance, and removes user group limits, enhancing performance for demanding workloads. Maximize Virtual Machine throughput: AZNFS now supports up to 256 TCP connections (up from16 in native NFS client) allowing throughput to reach VM NIC bandwidth (the maximum data transfer rate of virtual machine’s network interface card) or storage account limits. This benefits HPC workloads by ensuring high throughput for large dataset operations. Additionally, a small number (4 or fewer) of parallel file reads/writes can now fully saturate the VM NIC bandwidth even for larger VM sizes. Enhanced read/write speed: The updated AZNFS client outperforms native NFS client for read and write scenarios. For example, single file read/write performance is improved by a factor of 5x and 3x respectively, which can be useful for large database backup tasks requiring high single file throughput for writing and reading backup files. Refer to the link for a detailed performance comparison. Removal of the user's group limit: Linux NFS clients with a local Identity server can pass access permissions for up to 16 groups of a user, restricting resource access for users belonging to more than 16 groups. This update allows FUSE to handle permission checks, removing the 16-group limitation. Improved metadata query performance: READDIR can query more directory entries in one call. The Linux client has a limit of 1MB, whereas the updated AZNFS can now reach up to 3 MB. Customers with numerous files will experience quicker listing and metadata operations with reduced latency. This will be beneficial for workloads like EDA (Electronic Design Automation) and HPC (High Performance Computing) which often involve reading metadata for considerable number of files before selecting a subset for processing. Support for large file sizes (up to 5TB): The new release can support larger file sizes for sequential write patterns. Due to larger block sizes possible with AZNFS, users can create larger files up to the 5TB limit. With Linux clients, under best conditions, the max. file sizes were limited to ~3TB. CAD tools producing simulation and checkpoint data files over 3TB will benefit from this improvement. The following charts compare performance between updated AZNFS and the Native Linux client. Please refer the detailed benchmarks for more details. [Test parameters - VM: Standard D96ds v5, File size: 100GB, Linux NFS is with nconnect =16, Linux kernel 5.x.x ; Test used: DD test] Note: The VM supports higher read throughput than write throughput. For updated AZNFS, throughput starting from 4 parallel file read/write operations is constrained by VM NIC bandwidth, or it can scale higher. Getting Started Please register for the preview using this form. Please refer the link for instructions on how to install and use latest version of AZNFS. For any queries or feedback, please contact us at aznfs@microsoft.com. References: What is BlobFuse? - BlobFuse2 - Azure Storage | Microsoft Learn Network File System (NFS) 3.0 protocol support for Azure Blob Storage Mount Blob Storage by using the Network File System (NFS) 3.0 protocol on Linux Instructions to install and use latest version of AZNFS · Azure/AZNFS-mount Wiki
