ABFS Driver for Azure Blob Storage

We are excited to announce the release of the Azure Blob Filesystem (ABFS) driver with support for the Blob endpoint in flat namespace (FNS) storage accounts. This new capability is designed to help our analytics customers seamlessly migrate from the legacy WASB (Windows Azure Storage Blob) driver to the ABFS driver for accessing Azure Blob Storage. While we continue to recommend all big data and analytics customers to use Azure Data Lake Gen2 (ADLS Gen2) with the ABFS driver over the DFS endpoint, this new option should help customers transition to a supported scenario immediately, while they plan to ultimately move to ADLS Gen2 with HNS enabled accounts.

Note:
The WASB driver codebase has not received security updates or maintenance for an extended period and could render systems exposed to potential exploits. We strongly recommend all customers to prioritize immediate migration to the ABFS driver for Hadoop instances to protect critical data assets and infrastructure from emerging threats.

The Legacy WASB Driver and Its Deprecation

The legacy WASB driver is a Java-based library that enables Hadoop and Hadoop-like applications to persist data in a remote object storage account like Azure Blob storage via the Azure Blob Storage REST API. While WASB is still part of Apache Hadoop releases, it has not seen active development or maintenance for several years, which may impact its reliability and performance.

Microsoft's release of ADLS Gen2 included the ABFS driver, which is meant to replace the WASB driver. We collaborated with all our analytics platform partners to ensure a high-quality standard was achieved with ABFS

What's New in This Version of the ABFS Driver

The new version of the ABFS driver includes several enhancements and features designed to improve performance and usability for FNS accounts. Some of the key updates include:

• Support for Blob Endpoint in FNS Accounts: The ABFS driver now supports the Blob endpoint^# in FNS accounts, allowing customers to use the ABFS driver without needing to upgrade to HNS
• Improved Performance: The ABFS driver has been optimized to provide better performance compared to the WASB driver. For example, the ABFS driver makes significantly fewer metadata calls and write calls, resulting in faster data operations

#While the ABFS driver might function over the DFS endpoint in an FNS account, this configuration is not supported. We recommend that customers initialize the driver on the Blob endpoint when working with flat namespace (FNS) accounts.

• Enhanced AAD Based Authentication: The ABFS driver has a more robust support for AAD based authentication via oAuth 2.0 tokens. The oAuth 2.0 implementation includes native integration with Azure Identity libraries and a more sophisticated token acquisition, caching and renewal compared to what’s available in WASB.

Performance Comparison: WASB vs. ABFS in FNS

The ABFS driver demonstrated significant improvements over the WASB driver in FNS accounts. Here are some key highlights:

• Improved Transactions per Second (TPS): For common Read and Write operations, the ABFS driver has shown a median 55% and 20% improvement respectively compared to the WASB driver.
• Lesser Metadata Calls: The ABFS driver makes approximately 3.66 times fewer metadata calls compared to the WASB driver
• Fewer Write Calls: The ABFS driver makes about 1.66 times fewer write calls than the WASB driver
• Improved Runtime: The ABFS driver shows a significant reduction in runtime for various operations, making it more efficient for handling large datasets

These improvements highlight the enhanced performance and efficiency of the ABFS driver, making it a superior choice for customers currently using the WASB driver.

Migrating from WASB to ABFS Driver

For customers currently using the WASB driver, we recommend two migration options to transition to the ABFS driver:

1. Upgrade to HNS and Use ABFS Driver:
• The first and preferred option is to upgrade your storage accounts to HNS (hierarchical namespace) and use the ABFS driver. This option allows you to leverage the full capabilities of ADLS Gen2, including improved performance, security, and reliability. Upgrading to HNS enables the hierarchical structure, which is beneficial for managing large datasets and performing complex analytics operations.
• To upgrade your accounts to HNS, follow the detailed instructions provided here - Upgrading Azure Blob Storage to Azure Data Lake Storage | Microsoft Learn.
2. Remain in FNS and Use ABFS Driver:
• If upgrading to HNS is not immediately feasible, the second option is to remain in FNS configuration temporarily and move to using the new ABFS driver. With the new support for the Blob endpoint in FNS accounts, you can still benefit from the ABFS driver's enhanced performance and capabilities without changing your account configuration.

Call To Action

The release of the Azure Blob Filesystem (ABFS) driver with support for the Blob endpoint in flat namespace (FNS) accounts marks a significant milestone in our efforts to provide a seamless and efficient data storage solution for our customers. We encourage all customers currently using the WASB driver to migrate to the ABFS driver, either by upgrading their storage account to HNS or by remaining in FNS. By doing so, you can take advantage of the enhanced performance, security, and reliability offered by the driver.

If you have any questions on the driver or need assistance with the migration process, please drop us a note at askabfs@microsoft.com .

Reference:

ABFS Driver for Namespace Disabled Accounts (Flat Namespace)

Azure Blob Storage REST APIs (Blob Endpoint)

Hadoop Azure Commit List