Announcing the Public Preview of Metadata Caching for Azure Premium SMB File Shares
Azure Files is excited to announce the public preview of Metadata Caching for the premium SMB file share tier. Metadata Caching is an enhancement aimed at reducing metadata latency for file workloads running on Windows/Linux clients. In addition to lowering metadata latency, workloads will observe a consistent latency experience which will allow metadata intensive workloads to be more predictable and deterministic. Reduced metadata latency also translates to more data IOPS (reads/writes) and throughput. Once Metadata Caching is enabled, there is no additional cost or operational management overhead when using this feature. The following Metadata APIs will benefit from Metadata Caching. Create: Creating a new file; Up to 30% Faster Open: Opening a file; Up to 55% Faster Close: Closing a file; Up to 45% Faster Delete: Deleting a file; Up to 25% Faster Workloads that perform a high volume of metadata operations (creating/opening/closing/deleting) against a SMB Premium File share will receive the biggest benefit compared to workloads that are primarily data IO (e.g. databases) Example of metadata heavy workloads include: Web\App Services: Frequently accessed files for CMS\LMS services such as Moodle\WordPress. Indexing\Batch Jobs: Large scale processing using Azure Kubernetes or Azure Batch. Virtual Desktop Infrastructure: Azure Virtual Desktop\Citrix users with home directories or VDI applications with general purpose file system needs. Business Application: Custom line of business or legacy application with “Lift and shift” needs. CI\CD - DevOps Pipeline: Building, testing, and deployment workloads such as Jenkins open-source automation. Expected Performance Improvement with Metadata Cache. 2-3x Improved Metadata Latency Consistency Improved Metadata Latency beyond 30% Increased IOPS and Bandwidth up to 60% How to get started To begin onboarding to the Public Preview, please sign up on https://aka.ms/PremiumFilesMetadataCachingPreview and additional details will be provided. Regions Supported Australia East Brazil Southeast France South Germany West Central Switzerland North UAE North UAE Central US West Central (Updates will be provided as additional regions are supported and please sign up above to help influence the region prioritization) Who should Participate? Whether it is a new workload looking to leverage file shares or existing ones looking for improvements. Any workloads/usage patterns that contains metadata should be encouraged to onboard, specifically metadata heavy workloads that consist primarily of Create/Open/Close or Delete requests. To determine if your workload contains metadata, can use Azure Monitor to split the transactions by API dimension as described in the following article Thanks Azure Files TeamGeneral Availability: Azure Active Directory Kerberos with Azure Files for hybrid identities
We are excited to announce General Availability of Azure Files integration with Azure Active Directory (Azure AD) Kerberos for hybrid identities. With this release, identities in Azure AD can mount and access Azure file shares without the need for line-of-sight to an Active Directory domain controller.Accelerate metadata heavy workloads with Metadata Caching preview for Azure Premium Files SMB & REST
Azure Filespreviously announced the limitedpreview of Metadata caching highlighting improvements on the metadata latency (up to 55%) for workloads running on Azure Premium Files using SMB & REST. Now, we are excited to announce the unlimited public preview lighting up this capability on both new and existing shares in a broader set of regions.You can now automatically onboard your subscriptions to leverage this functionality using feature registration (AFEC) in supported regions. Feature Overview Metadata Caching is an enhancement aimed at reducing metadata latencyup to 55%for file workloads running on Windows/Linux environments. In addition to lower metadata latency, workloads will observe a2-3ximprovement in latency consistencymaking metadata intensive workloads more predictable and deterministic. Workloads that perform a high volume of metadata operations (e.g. AI/ML) will see the bigger benefit compared to workloads with high data IO (e.g. databases).Reduced metadata latency will also translate up to 3xincrease in metadata scale, andup to 60% increase in data IOPS (reads/writes) and throughput. Example of metadata heavy workloads include: Web\App Services: Frequently accessed files for CMS\LMS services such as Moodle\WordPress. Indexing\Batch Jobs: Large scale processing using Azure Kubernetes or Azure Batch. Virtual Desktop Infrastructure: Azure Virtual Desktop\Citrix users with home directories or VDI applications management needs. Business Application: Custom line of business or legacy application with “Lift and shift” needs. CI\CD DevOps Pipeline: Building, testing, and deployment workloads such as Jenkins open-source automation Building DevOps solutions using Metadata Caching Moodle deployment + Azure Premium Files with Metadata Caching Moodle consists of server hosting (cloud platforms), a database (MySQL, PostgreSQL), file storage (Azure Premium Files), and a PHP-based web server. It is used for course management (uploading materials, assignments, quizzes), user interaction (students accessing resources, submitting work, and discussions), and performance monitoring (tracking progress, reporting). Metadata Cache Benefit: Provides a faster andmore consistent user experience. GitHub Actions + Azure Premium Files with Metadata Caching GitHub Actions is an automation tool integrated with GitHub that allows developers to build, test, and deploy code directly from their repositories. It uses workflows, defined in YAML files, to automate tasks such as running tests, building software, or deploying applications. These workflows can be triggered by events like code pushes, pull requests, or scheduled times. Metadata Cache Benefit: Shorter build and deployment times when using Azure Premium Files with Metadata cache as the build artifact. How to get started To get started, registeryour subscription with the Metadata Cache feature using Azure portal or PowerShell. For Regional Availability please visit the following link Note:As we extend region support for the Metadata Cache feature, Premium File Storage Accounts in those regions will be automatically onboarded for all subscriptions registered with the Metadata Caching feature. Who should participate? Whether it is a new workload looking to leverage file shares or existing ones looking for improvements. Any workloads/usage patterns that contains metadata should be encouraged to onboard, specifically metadata heavy workloads that consist primarily of Create/Open/Close or Delete requests. To determine if your workload contains metadata, can use Azure Monitor to split the transactions by API dimension as described in the following article Thanks Azure Files Team For questions, please email:azfilespreview@microsoft.com
Azure File share NFS Snapshots is now Public Preview!!
Azure Files is a fully managed cloud file share service that enables organizations to share data across on-premises and cloud. The service is truly cross-platform and supports mounting of file share from any client that implements SMB and NFS protocols, it also exposes REST APIs for programmability. A key part file share service offering is its integrated backup for point in time recovery, this enables recovery of data from certain periods in the past in case data is deleted or corrupted. Such capability is best offered by Snapshots. We are excited to announce public preview of Snapshot support for NFS share. Customers using NFS shares will now be able to perform share level Snapshot management operations via REST API, PowerShell and CLI. Using Snapshots users will be able to roll back the entire filesystems or pull specific files that were accidentally deleted or corrupted. Therefore, it is always recommended to create a snapshot schedule that best suits your RPO (recovery point objective) requirement. Snapshot schedule frequency can be hourly, daily, weekly or monthly. Having such flexibility will help IT infra teams to serve a wide spectrum of RPO requirements suiting business needs. Although there are multiple scenarios where snapshots can benefit users, I will be highlighting two important scenarios that are widely sought after. Scenario #1 Recover files in case of accidental deletions, corruption, or user errors. Scenario #2Start-up read only replica of your application or database in few minutes to serve your reporting or analytics scenarios. Scenario #1 Recovery of data during accidental deletions and corruption is the most common scenario for admins during their day-to-day operations. There are solutions like backup (creating full and incremental copies of data) that help to recover the data from such scenarios, but snapshot technology offers more frequent recovery points (Lower RPO) to restore the data unlike backups. Snapshots are also considered to be space efficient since they capture only incremental changes. Creating snapshots of NFS file share is straightforward. This can be accomplished via Portal, REST, PowerShell or CLI. Let me show you how to access file share snapshots via NFS client to perform single file restore operations which can help you to recover data in accidental deletions or corruption scenarios. The first step is to mount the file share : cd “.snapshots” directory under root to view the snapshots that are already created. “.snapshot” directory is by default hidden but users will be able to access and read from the directory like a normal directory Each snapshot available/listed under .snapshot directory is a recovery point in itself. cd into the specific snapshot to view the files to be recovered. Initiate copy of required files and directories from snapshot to the desired location to complete the restore using cp command. Scenario #2 If you have an application or a database deployed on a NFS file share, one can create crash consistent or application consistent snapshot of NFS file share. Crash consistent is offered by default but Application consistent snapshots are not built-in capability, it will require admins to run few additional steps which can quiesce and unquiesce the application during snapshot creation process. For example, if you have a MySQL database one can write script to a execute a 3-step (quiesce(MySQL), snapshot(File share), and unquiesce(MySQL)) process to create an application-consistent snapshot of the database hosted on file share. Quiesce and unquiesce commands varies depending on the application or the database hosted on the file share. Such application consistent snapshots can be directly mounted on the desired NFS client and can be used as read only replicas for reporting and data analytics use-cases. The mounted snapshots can be used by applications or databases to create read-only static copies of the production database for analytics or reporting use-cases. They can be also copied to another location and then applications can be allowed to perform changes/writes. To improve copy performance, especially for large datasets with multiple files mount NFS snapshot using Nconnect setting which is available on latest Linux distributions and use fpsync to copy data out of snapshot to desired location. Sample scripts updated here For more information refer to documentation Mount an NFS Azure file share on Linux | Microsoft Learn Snapshot Share (FileREST API) - Azure Files | Microsoft LearnSoft delete for NFS Azure file shares is now Generally Available.
Soft deleteprotects your Azure file shares from accidental deletion. The following feature was already made available for SMB File share. Today, we are announcing the general availability of soft delete for NFS Azure file shares. The functionality will remain the same. Soft delete is like a recycle bin for your file shares. When a NFS file share is deleted, it transitions to a soft deleted state in the form of a soft deleted snapshot. You get to configure how long the soft deleted data is recoverable before it is permanently erased as part of the retention policy, by default its set to 7 days. Today,soft deleted NFS shares are not being counted towards the Storage account limit. We are providing a 30-day window (grace period) to update or change your automation scripts to account for soft deleted capacity when soft delete is enabled for NFS shares.By Sept 1st, 2024, we will roll out the change to start counting soft deleted capacity towards the account limit. What will change from September 1st 2024? soft deleted capacity will start counting towards storage account limits, this means the number of file shares that you can create on given storage account will be guarded by storage account limits when soft delete is enabled. Referherefor more information on supported capacity limits. Soft-deleted shares will be listed under deleted shares in the file share blade. To mount them or view their contents, you must undelete them. Upon undelete, the share will be recovered to its previous state, including all metadata as well as snapshots (Previous Versions). To successfully perform undelete ensure you do not have an active file share with the same name as that of deleted state. Soft delete feature is enabled by default on a storage account, the setting will apply to both NFS and SMB file shares. If you have an existing NFS file share in soft delete enabled account then, you would be enrolled for billing automatically. Soft delete enabled shares are billed on the used capacity when soft deleted at the snapshot rate while in the soft delete state. Billing would stop as soon the data is permanently deleted after the retention expires on the soft deleted state. Please refer here for Pricing and billing details.Storage migration: Combine Azure Storage Mover and Azure Data Box
Learn how to leverage the combination of Azure Storage Mover and Azure Data Box for offline migration + online catch-up. This post also discusses a popular migration strategy, helping you predict and minimize any downtime of workloads.
