Azure Storage Discovery now offers advanced capacity insights
Azure Storage Discovery is introducing advanced capacity insights to help customers gain deeper visibility into their data estate, identify inactive or inefficiently stored data, understand storage composition and retention considerations, and pinpoint resources for further investigation and optimization.Accelerated AI & Analytics workload on Azure Blob Storage: Up to 25x faster List Blobs operations
Today Azure Storage introduces in preview a new List Blobs optimization that accelerates listing operations by up to 25x with up to 15x lower client-side CPU utilization allowing customers to return millions of objects per second in List Blobs results. The list results are now returned in Apache Arrow format, a highly optimized and compact columnar response format that allows more efficient parsing and reduced client-side CPU utilization. Clients can now parallelize object list operations efficiently across multiple concurrent requests, while maintaining the same strong consistency of listing results that applications require. This increased performance and lower client CPU utilization is delivered within the existing List Blobs API via just a simple header change and is automatically invoked when using the updated Azure Blob Storage SDK’s. Why we built this Object storage was originally designed to provide low-cost, resilient data access through simple REST APIs. Early systems contained only a few million objects and were listed infrequently, so listing performance was not a priority. Cloud, big data, mobile, and cloud-native computing steadily increased data volumes and access demands. Since 2020, foundation models, LLMs, and large GPU fleets have pushed object storage to trillions of objects, making it an active data layer for AI. Modern AI and analytics workloads operates at the scale of trillions of objects. These workloads must repeatedly discover and inventory vast datasets for pre-training, analytics, fine-tuning, and inference, making frequent listing operations a significant source of storage-system pressure and client-side CPU consumption. Downstream AI training and data analytics jobs are gated by the time required to enumerate immense datasets, while parsing the results consumes client-side CPU that could otherwise run the workload. To meet these rapidly growing demands of AI workloads, we’ve built the next generation of Azure Blob Storage listing capabilities that deliver the performance required at this massive scale. Next, we will dive deeper into the details of how List Blobs performance and scalability have been accelerated and how simple it is for customers to take advantage of this new level of performance. Faster, more efficient, listing results with lower client CPU utilization The Apache Arrow format was selected as the most efficient new way to deliver the performance and scalability gains required for efficiently listing millions of objects per second. Apache Arrow is an open-source, compact columnar format that returns List Blobs results in an optimized response roughly one third the size of the XML format used by most cloud object-storage listing APIs, making parsing faster and easier. The new format is enabled with a simple request-header change. The List Blobs API then packages and accelerates results automatically while preserving strong consistency, so newly written objects remain immediately visible. Because Apache Arrow is compact and efficient to parse, client-side CPU utilization per object listed decreased by up to 15x, freeing compute resources for the workload itself. Submitting multiple List Blobs requests concurrently further improves performance, delivering up to a 25x increase in enumeration speed with a single request-header change, while preserving schema consistency and compatibility with existing XML responses. This new List Blobs performance enhancement via Apache Arrow remains an additive, opt-in extension of the existing List Blobs API, not a replacement. The existing XML-based List Blobs API remains unchanged, and current clients not adopting the new header continue to work with no breaking changes, but without the performance and lower CPU-utilization benefits. The next figure shows the comparison of a parallelized listing operation using Apache Arrow format compared with the XML baseline on 16 clients and 48 threads per client. rclone accelerates listing of 100K objects from 24 seconds to 1.1 seconds rclone is a popular command-line program to manage, copy, move and replicate files on and between cloud storage destinations. rclone is a widely used open-source tool for moving and syncing data across almost all types of cloud storage. Listing is a critical part of rclone’s synchronization workflow. To perform these data management operations at scale for large numbers of objects, rclone must enumerate the objects that are intended to be copied, moved, or replicated. For example, before and after syncing Azure Blob Storage containers, rclone performs a large-scale listing to compare the source and target states. After updating rclone to incorporate the simple header change for the enhanced Apache Arrow-based List Blobs API calls, rclone was able to reduce their List Blobs time-to-completion by 21.7x from 24 seconds to 1.1 seconds on 100K object datasets. The chart below shows rclone's measured wall clock time for listing a single container with 100,000 entries (columns, left axis) alongside the resulting speedup versus the classic XML path (line, right axis). Two things stand out in the results: Apache Arrow is an immediate win on its own: With no parallelism at all, sequential Arrow listing is 3.5× faster than the current XML-based List Blobs results, reducing listing operation time to completion from 23.9 seconds to 6.9 seconds . Parallel enumeration compounds the gains: Throughput climbs steadily with concurrency, reaching a 21.7× speedup at a parallelism of 30 and completing the same listing of 100,000 entries in just 1.1 seconds. The accelerated List Blobs results are returned with the same consistency using the same List Blobs API but now returned much faster with no breaking changes for existing clients. That is exactly the outcome we set out to deliver with this new capability. In rclone's own words about these results: " rclone has to list containers before it can sync them; with Apache Arrow and parallelism enabled this will make a sync of a directory with millions of files get going 20x faster. The Azure Storage team has been very responsive to our feedback during the preview which made the integration straightforward. The new Go SDK works very well and required very few code changes. Our Azure Blob Storage users are going to love this!" Nick Craig-Wood, rclone Lead Developer To get started, you can download rclone from the official rclone website. If you are running rclone v1.74.0 or later you can enable the Apache Arrow listing with the --azureblob-use-arrow-list flag and enable listing parallelism with --azureblob-list-parallelism. As described in the testing, if you set “--azureblob-list-parallelism 30” this will get you the most performance listing from Azure with Arrow listing also enabled. A sincere thank you to the rclone community for adopting List Blobs with Apache Arrow early and sharing such clear, quantified results. Feedback like that is invaluable as we advance toward general availability. Happy listing rcloners! How to get started At the REST layer, the Arrow response is negotiated with the Accept: application/vnd.apache.arrow.stream header on a minimal x-ms-version of 2026-06-06 or later. This will return a response content that will be an Apache Arrow IPC stream that can be decoded and used to instantiate a RecordBatchStreamReader using Apache Arrow SDKs in any language. An example of decoding from Rest API response is provided below using Apache Arrow Python SDK for decoding: table = pa.ipc.open_stream(resp.content).read_all() print(table.schema) print("\nrows:", table.num_rows, " columns:", table.num_columns) Name: string not null Creation-Time: timestamp[s] Last-Modified: timestamp[s] BlobType: string ResourceType: string not null Etag: string Content-Length: uint64 Content-Type: string Content-MD5: string AccessTier: string AccessTierInferred: bool LeaseState: string LeaseStatus: string ServerEncrypted: bool -- schema metadata -- NumberOfRecords: '100' NextMarker: '' rows: 100 columns: 14 import pandas as pd df = table.to_pandas() df[["Name", "BlobType", "Content-Length", "size_mb", "AccessTier"]].head(6) Name BlobType Content-Length AccessTier 0 train_chunk10_shard1.jsonl.zst BlockBlob 215 Hot 1 train_chunk10_shard10.jsonl.zst BlockBlob 216 Hot 2 train_chunk10_shard2.jsonl.zst BlockBlob 215 Hot 3 train_chunk10_shard3.jsonl.zst BlockBlob 215 Hot 4 train_chunk10_shard4.jsonl.zst BlockBlob 215 Hot 5 train_chunk10_shard5.jsonl.zst BlockBlob 217 Hot This new accelerated performance for List Blobs listing via Apache Arrow can be transparently enabled on Python, Java, .NET, C++ and Go SDKs with a simple option in the container listing function. Enabling via Azure Blob SDKs allows the performance benefits to be achieved without changing the listing interface and returned object formats in the Azure Storage SDK. This new accelerated performance for List Blobs is available in public preview across the Azure Storage client libraries listed below. To evaluate the capability, use the corresponding minimum preview version for your preferred language. SDK Minimum version (preview) .NET 12.30.0-beta.1 Python 12.31.0b1 Java 12.36.0-beta.1 Go v1.8.1-beta.1 C++ 12.19.0-beta.1 JavaScript 12.34.0-beta.1 To get started, update to a preview enabled SDK for your language and start testing the new feature with our samples, or add the new listing options to your existing REST calls Parallelizing listing operations unlocks the double-digit performance gains described in this article. The optimal strategy depends on the namespace layout and distributes listing requests across multiple threads. The two most common approaches are: Use delimiter parameter to recursively fan out additional threads for each BlobPrefix, walking the namespace. Partition the namespace with startFrom and endBefore, then process the ranges across multiple threads. This is the approach used by rclone. The best approach depends on the specific namespace layout and can be optimized by tuning both the algorithm and the level of parallelism. Limitations This new Apache Arrow-powered performance optimization for List Blobs is today supported on flat namespace (FNS) Azure Blob Storage accounts. In scenarios where the account is Hierarchical Namespace (HNS) enabled, if the List Blobs REST API is called with the new header, it will return a 409 (Conflict) error code. This error code can be used to fallback on the client application to standard XML listing. Public preview is where your input shapes the product. Try it on your largest containers, tell us what you measure, and let us know what would make it even better through this form. References List Blobs (REST API) - Azure Storage | Microsoft Learn List Blobs with Apache Arrow SamplesAHEAD helps us launch the Strategic Azure Storage Services Partner Program
AHEAD becomes the first Azure Storage Strategic Channel Partner by demonstrating their expertise in helping customers select the ideal Azure Storage, or Azure Storage ISV, Service to offer the ideal price / performance solution for their application and helping customers to migrate to Azure quickly and safely.Enterprise Identity Meets Secure File Transfer: Entra ID Public Preview on Azure Blob Storage SFTP
Updated on June 23 2026: Microsoft Entra ID-based access for Azure Blob Storage SFTP is now Generally Available. GA enhancements: Improved ABAC by resolving inconsistencies with the Storage Blob Data Owner role (eliminating timeout issues) while adding support for sub-operations to enable more granular access control. Learn more here. We are excited to announce the public preview of Entra ID-based access for Azure Blob Storage SFTP. This new capability enables you to use Microsoft Entra ID (formerly Azure Active Directory) identities (including guest users via Entra External Identities) to securely connect to Azure Blob Storage via SFTP without needing local users. This feature eliminates the operational overhead of managing local SFTP users and passwords by introducing enterprise-grade identity management powered by Microsoft Entra ID. For IT administrators and security teams, this means no more creating, tracking, rotating, or decommissioning local SFTP credentials. For developers and architects, it means seamless integration with your existing identity infrastructure. For business users, it means faster, more secure access to the data they need, all while maintaining compliance with enterprise security policies. Azure Blob Storage SFTP Azure Blob Storage SFTP natively enables secure file access and management without third-party solutions. This simplifies operations for customers and removes the need for complex, custom SFTP solutions. Until this Public Preview, Azure Blob Storage SFTP utilized a form of identity management called local users as the only authorization mechanism. Local users must use either a password or a Secure Shell (SSH) private key credential for authentication. Learn more about local users here. The Challenge: SFTP Local User Management Organizations currently face challenges when managing SFTP access at scale with Azure Storage SFTP Local Users. Local User based SFTP access require IT teams to: Manually create and provision local user accounts for each SFTP user Generate, distribute, and securely store SSH keys or passwords Implement custom workflows for lifecycle management Manage offboarding processes to ensure departed users lose access immediately Audit and track access across disconnected identity silos Handle external partner and vendor access through ad-hoc, often insecure methods The Solution: Enterprise Identity Meets Secure File Transfer With Entra ID-based access for Azure Blob Storage SFTP, you can now leverage your organization's centralized identity platform to authenticate and authorize SFTP users. This integration brings the full power of Microsoft Entra ID to your file transfer workflows, delivering the following benefits: 1. Eliminate Local User Management Simplify SFTP management by assigning access with Entra ID—no separate SFTP accounts needed. No local credential generation or distribution—users authenticate with their existing corporate credentials No orphaned accounts when users change roles or leave the organization Reduced attack surface by eliminating static, long-lived local credentials Centralized user lifecycle management through your existing identity platform 2. Enterprise-Grade Identity and Security Leverage the full security capabilities of Microsoft Entra ID for your SFTP infrastructure: Multi-Factor Authentication (MFA): Require additional verification factors beyond passwords, significantly reducing the risk of account compromise Conditional Access: Define policies that grant or block access based on user location, device compliance, sign-in risk, and other conditions Identity Protection: Benefit from Microsoft threat intelligence and risk detection to identify and respond to compromised accounts Privileged Identity Management (PIM): Provide just-in-time elevated access for administrative operations 3. Native Azure RBAC, ABAC, and ACL Integration Your SFTP access control seamlessly integrates with Azure comprehensive authorization framework: Role-Based Access Control (RBAC): Assign built-in or custom roles at the storage account, container, or even blob level Attribute-Based Access Control (ABAC): Create sophisticated access policies based on resource tags, user attributes, and environmental conditions Access Control Lists (ACLs): Apply fine-grained permissions at the directory and file level for hierarchical namespace-enabled accounts Unified Permission Model: SFTP access respects the same permissions as REST API, Azure CLI, and other access methods—no separate permission system to manage 4. Faster SFTP Onboarding and Time-to-Value Onboard new SFTP users or partners in minutes instead of hours or days, saving significant time and boosting business agility. 5. Secure External Collaboration with Entra External Identities Seamlessly enable secure external SFTP access by allowing partners to authenticate with their own credentials using Entra External Identities (Azure AD B2B). External users authenticate with credentials they already manage Full audit trail of external user activity Ability to apply Conditional Access policies to external users Automatic access revocation when B2B relationships end Real World Scenarios Financial Services: A bank receives daily transaction files from merchants via SFTP. Merchants authenticate with their own Entra ID credentials (B2B collaboration), MFA is enforced, and access is restricted to assigned directories. Access is instantly revoked when a merchant is removed from the B2B directory. Healthcare: A hospital exchanges patient data with insurers and labs. Entra ID authentication ensures only authorized staff access sensitive PII, with full audit logs for HIPAA compliance. Conditional Access restricts connections to approved locations and devices. Media & Entertainment: A production company enables freelance editors and agencies to transfer large media files. Entra External Identities provide time-limited access and automatic revocation when projects end—no need for local SFTP accounts. Manufacturing: A manufacturer receives CAD files and orders from suppliers using SFTP. With Entra ID, suppliers use unified credentials and access policies across all systems, streamlining supply chain management. How It Works Entra ID simplifies SFTP access to Azure Blob Storage by authenticating users with their corporate credentials. After authentication, users receive a short-lived Open SSH certificate to connect. The service verifies certificate validity and user permissions, enabling secure file operations and automatic access revocation in line with current identity policies. Learn more here. Getting started with the Public Preview We encourage you to try Entra ID-based access for Azure Blob Storage SFTP in your non-production environments today. Learn more about how to register for the preview and get started with the detailed ms docs learn guide here. This preview gives you an opportunity to shape the feature development by providing feedback on what works well and what could be improved. Note: Local user accounts for SFTP access are still supported, but we strongly recommend switching to Entra ID-based access for greater security, simpler management, and automatic access control. Questions or feedback? We would love to hear from you! Reach out to our team at blobsftp@microsoft.com We are excited to bring enterprise-grade identity management to Azure Blob Storage SFTP, and we cannot wait to see how you use this capability to simplify operations, enhance security, and enable new collaboration scenarios. Happy transferring!
Transforming Data migration using Azure Copilot
Introduction Data migration is critical, yet it is one of the most complex tasks in any cloud adoption journey. Whether you’re moving workloads from on-premises environments, consolidating hybrid deployments, or transitioning from other cloud providers, the migration process involves multiple tools, intricate planning, and risk management. What’s New in Azure Copilot With the new “Storage Migration Solutions Advisor” capability in Azure Copilot, Microsoft is transforming this experience into a conversational, AI-driven workflow that accelerates decision-making and reduces operational friction. Why This Matters Traditionally, customers faced challenges such as: Weeks of advisory time spent choosing the right migration tool amongst the many (Azure Storage Mover, AzCopy, Data Box, File Sync etc., and various Partner solutions). High support overhead due to missteps during migration if a sub-optimal tool or service is used. The Storage Migration Solutions Advisor feature introduces: Conversational Guidance: Share your migration needs with Copilot, like talking with an Azure advisor. Scenario-Based Recommendations: Tailored suggestions based on transfer data size, protocol, and bandwidth. Expanded Coverage: Supports on-premises to Azure, cloud-to-cloud (AWS/GCP to Azure), and hybrid scenarios. Native and Partner solutions: Copilot can recommend Microsoft-native (1P) solutions and third-party (3P) tools for specialized scenarios —ensuring flexibility for enterprise needs. User Workflow: Step-by-Step Initiate Migration: Start with a prompt like “How can I migrate my data into Azure?” or “What’s the best tool for moving 1 PB from AWS S3 to Azure Blob?” Provide Details: Copilot will guide you by asking for details about your requirement, such as source type (e.g., NAS, SAN, AWS S3, GCS), protocol (e.g., NFS, SMB, S3 API), target (e.g., Azure Blob, Files, Elastic SAN), data size, and bandwidth. Azure and Partner Solutions: Based on your requirements, Copilot recommends the best-fit Azure solution. If a partner solution is better suited to your requirement, Copilot will also select and recommend the appropriate solution with links to its documentation and/or its Azure marketplace page. Examples Copilot generates recommendations for migrating an on-premises file share to Azure Files. Figure 1 Prompt from user invokes Copilot Migration recommendation workflow Figure 2 Copilot understanding protocols that customer environment has access to Figure 3 Copilot asking user's target Storage type Figure 4 Copilot gathering inputs on data size, network bandwidth availability and transfer direction Figure 5 Copilot recommendation for user scenario Copilot recommends Partner solutions for specialized migration scenarios Figure 1 Prompt from user invokes Copilot Migration recommendation workflow Figure 2 Copilot understanding protocols that customer environment has access to Figure 3 Copilot asking user's target Storage type Figure 4 Copilot gathering inputs on data size, network bandwidth availability and transfer direction Figure 5 Copilot recommendation for user scenario Pro Tips Run a small proof-of-concept migration to estimate throughput and timing, especially for large datasets or small file sizes. Combine Copilot’s recommendations with Azure Storage Discovery for visibility into your storage estate after migration. Getting Started Navigate to Azure Portal → Copilot. Try prompts like: o “Help me migrate an NFS share to Azure Files.” o “What’s the best tool for moving 1 PB from AWS S3 to Azure Blob?” Explore Manage and migrate storage accounts using Azure Copilot | Microsoft Learn for detailed guidance. Ready to simplify your migration journey? Start using Azure Copilot’s Storage Migration Solutions Advisor today and experience AI-driven efficiency for your cloud transformation.
Unlocking Storage Optimizations: Smart Tiering for Blobs and ADLS in Azure Storage
We are excited to introduce the public preview of smart tier for Azure Blob and Azure Data Lake Storage. Smart tier is a fully managed, automated data tiering solution that takes the guesswork and manual effort out of optimizing your storage costs. Smart tier continuously analyzes your data’s access patterns and automatically moves objects between the hot, cool, and cold tiers. Smart tier will keep regularly accessed objects on the hot capacity tier to optimize transaction costs and moves inactive objects after 30 days to the cool tier capacity tier and after an additional 60 days of inactivity to the cold capacity tier. If you access an object in cool or cold tiers again, it’s instantly promoted back to the hot tier, restarting the cycle. This ensures your data is always in the most cost-effective tier with zero manual intervention, making it the ideal online tier for datasets with mixed or unknown access patterns. Getting started Using smart tier is quick and easy: Enabling smart tier is simple: Just select smart tier as the default access tier through the storage account configuration for any storage account with zonal redundancy. Smart tier is available in all zonal public cloud regions, supporting both flat and hierarchical namespaces. Billing is straightforward: You will pay the regular hot, cool, and cold capacity rates, with no extra charges for tier transitions, early deletion, or data retrieval. Even moving existing objects into smart tier does not incur tier change fees. There’s just a small monitoring fee for the orchestration. Smart tier is configured at the account level. It can be configured via API or the Azure portal as the default access tier setting for new and existing storage accounts. Existing objects following the default access tier setting from the account will be moved to smart tier automatically. Objects that are explicitly tiered, i.e. to the hot tier, will remain in the same account and will not be moved to other capacity tiers. Smart tier will always keep small objects that are below 128 KiB in size in the hot capacity tier for efficiency and those objects will not incur a monitoring charge. If objects below 128 KiB increase in size, the smart tiering patterns apply for those objects as well. The automatic down tiering of inactive data, paired with the billing model simplifications of Smart tier can lead to large cost savings over time. In the metrics view of the storage account you can see the distribution across the capacity tiers for smart tiered objects by both object count and capacity. This account shows smart tier in action, moving inactive objects to the cool and cold capacity tier, thereby drastically reducing the capacity charges without any manual intervention. "2 years ago, Qumulo partnered with Microsoft to deliver the first truly elastic, unlimited capacity, fully managed file system, Azure Native Qumulo, which was built on Azure Blob," said Brandon Whitelaw SVP of Product at Qumulo. "Qumulo shared feedback with Microsoft on our ideal solution for data tiering and Microsoft clearly delivered, meeting all expectations. With today's smart tier announcement, Qumulo will immediately enhance our offerings with these new capabilities, delivering greater functionality and control over data lifecycle management. We are thrilled with the feature set Azure is delivering at launch” Note that smart tier is not supported with append and page blobs. Smart tier is the ideal tier to choose when you are looking to store your data on standard online tiers but are not fully aware of the data access patterns or do not want to manage data transitions across online tiers. Objects managed by smart tier are not subject to lifecycle management policies, ensuring that automated tiering decisions are based solely on access patterns. Smart tier for block blobs is now available in public preview for both Azure Blob Storage and Azure Data Lake Storage for storage accounts with zonal redundancies, including ZRS, GZRS and RA-GZRS. Unlock cost savings by adding smart tier to your blob storage accounts in one easy step: https://aka.ms/BlobSmarttier. Please reach out to us for any feedback or questions, we would love to hear from you: smartblob@microsoft.com
Manage/restore metadata when blob is updated
I'm using an Azure Storage container of block blobs as a data source for an Azure AI Search Index, and I'm using Blob metadata key value pairs for some custom data. But metadata gets wiped when a blob is updated. How are folks managing that? For reference, I've got a CosmosDB set up also for now with a cross-reference I can restore from, but it's manual. I considered using Cosmos as my data source instead, but I also need a place to store/serve media files from related to these records.
Priority Replication for Geo Redundant Storage and Object Replication is Generally Available
We are excited to announce the General Availability of priority replication for Geo Redundant Storage (GRS) and Object Replication (OR). Priority replication enhances the replication process of GRS/GZRS and OR, ensuring guaranteed 15 minute synchronization between regions and improved replication times. Enabling priority replication will also provide an official Service Level Agreement (SLA) for all user workloads that meet the SLA criteria. What is Geo Priority Replication? Azure Storage has offered users the choice of geo-redundant (GRS) or geo-zone-redundant (GZRS) replication for their storage accounts for several years. As displayed in the diagram below, with GRS/GZRS, data in the storage account is asynchronously replicated from the primary region to the secondary region: Since the data is asynchronously replicated, in the event there is a disaster in the primary region, and an unplanned failover is initiated, there is a possibility of some data loss. The Last Sync Time (LST) property of a GRS/GZRS account is currently used to provide users with the recovery point objective (RPO) of their account. The LST of the account indicates the most recent time that data from the primary region is guaranteed to have been written to the secondary region. All data and metadata written prior to the LST is guaranteed to be available on the secondary. However, any data or metadata written after the LST may not have been replicated to the secondary and could be lost in the event of a disaster impacting the primary region or an unplanned failover. Azure Blob Storage is now introducing Geo priority replication which will provide an SLA guarantee for the LST/RPO of Block Blob data in GRS/GZRS accounts. Geo priority replication enhances the replication process of GRS/GZRS storage accounts allowing for accelerated replication between the primary and secondary regions. The SLA guarantees the Last Sync Time of Block Blob data will be 15 minutes or less 99.0% of the billing month. With this guaranteed sync time, users can feel more confident about their data’s durability and availability, especially if there is an unexpected outage and failover in the primary region. Please refer to the official SLA Terms for a comprehensive list of eligibility requirements. Geo Priority Replication supports the following SKUs: GRS, RA-GRS, GZRS and RA-GZRS for Block Blob data. What is Object Replication Priority Replication? Without priority replication, object replication asynchronously copies all operations from a source storage account to one or more destination accounts, but completion time is not guaranteed. Object replication priority replication will allow users to obtain prioritized replication from the source storage account to the destination storage account of their replication policy. When you enable priority replication, you can benefit from the associated Service Level Agreement if your source and destination account are located within the same continent. The SLA will guarantee that 99.0% of operations are replicated from the source container to the destination container of their OR policy within 15 minutes for the billing month. Please refer to the official SLA Terms for a comprehensive list of eligibility requirements. How to monitor SLA compliance for Geo and OR Priority Replication? Geo Priority Replication: Geo priority replication introduces a new metric, Geo Blob Lag. This metric allows users to monitor their Blob Replication lag, the number of seconds since the last full data synchronization between the primary and secondary region. This will allow users to track the SLA compliance of their GRS/GZRS account. If an account’s Geo Blob Lag remains within 900 seconds (15 minutes), they can confirm the data replication is within SLA. To learn more about the Geo Blob Lag metric view, Geo Blob Lag Metric. Object Replication Priority Replication: Replication Metrics for Object Replication is now Generally Available. These metrics empower users to troubleshoot replication delays and will help users with priority replication enabled monitor their SLA compliance. Metrics now supported are: Pending Operations: Tracks the total number of operations pending replication from the source to the destination storage account of your OR policy Pending Bytes: Tracks the total volume of data pending replication from the source to the destination storage account of your OR policy These metrics are grouped into various time buckets including 0-5 minutes, 10-15 minutes and > 24 hours. Users with OR priority replication that would like to ensure all their operations are replicating within 15 minutes; can monitor the larger time buckets (ex: 30 mins – 2hours or 8-24 hours) and ensure they are at zero or near zero. Users also have other options such as checking the replication status of their source blob. Users can check the replication status of a source blob to determine whether replication to the destination has been completed. Once the replication status is marked as “Completed,” the user can guarantee the blob is available in the destination account. How to get started? Getting started is simple, to learn more about the step-by-step process to opt-in to Geo priority replication: Azure Storage Geo Priority Replication - Azure Storage | Microsoft Learn. Or if you would like to learn about the step-by-step process to enable OR Priority Replication, view: Object Replication Priority Replication - Azure Storage | Microsoft Learn. Feedback If you have questions or feedback, reach out at priorityreplication@microsoft.com.Take Data Management to the next level with Silk Software-Defined Azure Storage
Note: This article is co-authored by our partner Silk. In today’s data-driven world, every enterprise is under pressure to make smarter decisions faster. Whether you're running production databases, training machine learning models, running advanced analytics, or building customer-facing applications, your data needs to be more agile, secure, and readily accessible than ever before. That’s where Silk’s software-defined cloud storage platform on Microsoft Azure comes into play — bringing performance, resiliency, and intelligence to data management across the cloud. With the recent addition of Silk Echo, you can now supercharge your Copy Data Management (CDM) strategy to ensure your data isn’t just protected, it’s available instantly for any purpose — a true strategic asset. Transforming Azure IaaS with Silk's Platform Microsoft Azure offers a rich ecosystem of services to support every stage of your cloud journey, and when paired with Silk, customers gain a game-changing storage and data-services layer purpose-built for performance-intensive workloads. Silk’s software-defined storage platform runs on Azure infrastructure as a high-performance data layer between Azure compute and native storage. It works by orchestrating redundant sets of resources, as close to the DB compute as possible. Aggregating and accelerating the native capabilities of Azure, enabling databases to reach the maximum physical limits of the underlying hardware. Silk makes use of the excellent L-series of storage optimized VMs and NVME media, ensuring the data is always quickly available and able to withstand multiple failures using erasure coding. Silk is designed to address common challenges customers face when migrating relational databases — such as SQL Server, Oracle, and DB2 — to the cloud: Performance Bottlenecks: On-prem workloads often rely on ultra-low latency, high-throughput storage systems with features that are difficult to replicate in the cloud. Data Copy Sprawl: Multiple non-production environments (dev, test, QA, analytics) mean many redundant data copies, leading to storage inefficiencies. Operational Overhead: Managing snapshots, backups, and refreshes across environments consumes time and resources. Silk changes the game with: Extreme performance in the Azure cloud. Up to 34GB/s of throughput from a single VM. The combination of Silk and Azure provide a unique cost/performance balance, through the combination of a very low latency software defined cloud storage platform and sharing of Azure resources. Inline data deduplication and compression for optimized resource utilization. Autonomous, fully integrated, non-disruptive, zero cost snapshots and clones for effortless environment refreshes. Multi-zone resilience and no single point of failure. This makes it easier than ever to lift and shift critical applications to Azure, with the confidence of consistent performance, uptime, and flexibility. Elevating CDM with Silk Echo for AI While Silk’s core platform solves performance and efficiency challenges, Silk Echo for AI introduces an intelligent, AI-powered layer that revolutionizes how organizations manage and leverage their data across the enterprise. At its core, Silk Echo for AI offers next generation Copy Data Management capabilities that empower IT teams to accelerate digital initiatives, reduce costs, and maximize the value of every data copy. Key Benefits of Silk Echo for AI Smarter Data Copying and Cloning Silk Echo leverages AI to understand data access patterns and recommends the optimal strategy for creating and managing data copies. Instead of manually managing snapshots, you can automate the entire workflow — ensuring the right data is in the right place at the right time. Instant, Space-Efficient Clones Using Silk’s advanced snapshot and cloning engine, Echo creates fully functional clones in seconds, consuming minimal additional storage resources. Teams can spin up dev/test environments instantly, accelerating release cycles and experimentation. Cross-Environment Data Consistency Echo ensures consistency across copies — whether you're cloning for testing, backup, or analytics — and with AI-driven monitoring, it can detect drift between environments and recommend synchronizations. Policy-Based Lifecycle Management Define policies for how long data copies should live, when to refresh them, and who has access. Echo automates the enforcement of these policies, reducing human error and ensuring compliance. Optimized Resource Consumption Silk Echo minimizes redundant data storage through smart deduplication, compression, and AI-driven provisioning — resulting in cost savings of 50% or more across large-scale environments. Enablement for AI/ML Workflows For data science teams, Silk Echo provides curated, up-to-date data clones without impacting production environments — essential for model training, experimentation, and validation. Real-World Use Case: Streamlining Dev/Test and AI Pipelines Consider Sentara Health, a healthcare provider migrating their EHR and SQL Server workloads to Azure. Before Silk, environment refreshes were time-consuming, often taking days or even weeks. With Silk Echo for AI, the same tasks will be completed in minutes. Now, development teams have self-service access to fresh clones of production data — enabling faster iteration and better testing outcomes. Meanwhile, their data science team leverages Echo’s snapshot automation to feed AI models with real-time, production-grade data clones without risking downtime or data corruption. All of this runs seamlessly on Azure, with Silk ensuring high performance and resilience at every step. Joint Value of Silk and Microsoft Together, Silk and Microsoft are unlocking a new level of agility and intelligence for enterprise data management: Data-as-a-Service: Give every team — DevOps, DataOps, AI/ML — access to the data they need, when they need it. Free snapshots democratize data so up-to-date copies can be made quickly for any team member who can benefit from it. AI-Ready Database Infrastructure: Your infrastructure evolves from a reactive model which is addressing problems as they arise (i.e. triggering responses on alerts), to a predictive model that utilizes AI/ML to forecast issues and mitigate them before they occur by learning patterns of behavior. Silk enables real-time AI inferencing, for business-critical agents that require access to up-to-date operational data. Reduced Costs, Improved ROI: Storage optimization, reduced manual overhead, and faster time to value — backed by Azure’s scalability and Silk’s performance. Accelerated Cloud Migrations: Achieve the enhanced scalability and flexibility of a cloud migration for your Tier 1 databases without refactoring. Get Started Ready to take your data management strategy to the next level? Explore how Silk’s software-defined storage and Silk Echo for AI can accelerate your transformation on Microsoft Azure. Whether you're modernizing legacy systems, building AI-driven applications, or simply trying to get more value from your cloud investments, Silk and Microsoft are here to help. By embracing the power of Silk’s software-defined storage and Silk Echo, organizations can finally make their data in the cloud work smarter, not harder. Contact Alliances@silk.us for a deeper dive on Silk!Beyond Basics: Practical scenarios with Azure Storage Actions
If you are new to Azure Storage Actions, check out our GA announcement blog for an introduction. This post is for cloud architects, data engineers, and IT admins who want to automate and optimize data governance at scale. The Challenge: Modern Data Management at Scale As organizations generate more data than ever, managing that data efficiently and securely is a growing challenge. Manual scripts, periodic audits, and ad-hoc cleanups can’t keep up with the scale, complexity, and compliance demands of today’s cloud workloads. Teams need automation that’s reliable, scalable, and easy to maintain. Azure Storage Actions delivers on this need by enabling policy-driven automation for your storage accounts. With Storage Actions, you can: Automate compliance (e.g., legal holds, retention) Optimize storage costs (e.g., auto-tiering, expiry) Reduce operational overhead (no more custom cleanup scripts) Improve data discoverability (tagging, labeling) Real-World Scenarios: Unlocking the Power of Storage Actions Let’s explore 3 practical scenarios where Storage Actions can transform customers’ data management approach. For each, we’ll look at the business problem, the traditional approach, and how Storage Actions makes it easier with the exact conditions and operations which can be used. Scenario 1: Content Lifecycle for Brand Teams Business Problem: Brand and marketing teams manage large volumes of creative assets - videos, design files, campaign materials that evolve through multiple stages and often carry licensing restrictions. These assets need to be retained, frozen, or archived based on their lifecycle and usage rights. Traditionally, teams rely on scripts or manual workflows to manage this, which can be error-prone, slow, and difficult to scale. How Storage Actions Helps: Azure Storage Actions enables brand teams to automate the content lifecycle management using blob metadata and / or index tag. With a single task definition using an IF and ELSE structure, teams can apply different operations to blobs based on their stage, licensing status, and age without writing or maintaining scripts. Example in Practice: Let’s say a brand team manages thousands of creative assets videos, design files, campaign materials each tagged with blob metadata that reflects its lifecycle stage and licensing status. For instance: Assets that are ready for public use are tagged with asset-stage = final Licensed or restricted-use content is tagged with usage-rights = restricted Over time, these assets accumulate in your storage account, and you need a way to: Ensure that licensed content is protected from accidental deletion or modification Archive older final assets to reduce storage costs Apply these rules automatically, without relying on scripts or manual reviews With Azure Storage Actions, the team can define a single task that evaluates each blob and applies the appropriate operation using a simple IF and ELSE structure: IF: - Metadata.Value["asset-stage"] equals "final" - AND Metadata.Value["usage-rights"] equals “restricted” - AND creationTime < 60d THEN: - SetBlobLegalHold: This locks the blob to prevent deletion or modification, ensuring compliance with licensing agreements. - SetBlobTier to Archive: This moves the blob to the Archive tier, significantly reducing storage costs for older content that is rarely accessed. ELSE - SetBlobTier to Cool: If the blob does not meet the above criteria whether it’s a draft, unlicensed, or recently created, it is moved to the Cool tier. Once this Storage Action is created and assigned to a storage account, it is scheduled to run automatically every week. During each scheduled run, the task evaluates every blob in the target container or account. For each blob, it checks if the asset is marked as final, tagged with usage-rights, and older than 60 days. If all these conditions are met, the blob is locked with a legal hold to prevent accidental deletion and then archived to optimize storage costs. If the blob does not meet all of these criteria, it is moved to the Cool tier, ensuring it remains accessible but stored more economically. This weekly automation ensures that every asset is managed appropriately based on its metadata, without requiring manual intervention or custom scripts. Scenario 2: Audit-Proof Model Training Business Problem: In machine learning workflows, ensuring the integrity and reproducibility of training data is critical especially when models influence regulated decisions in sectors like automotive, finance, healthcare, or legal compliance. Months or even years after a model is deployed, auditors or regulators may request proof that the training data used has not been altered since the model was built. Traditionally, teams try to preserve training datasets by duplicating them into backup storage, applying naming conventions, and manually restricting access. These methods are error-prone, hard to enforce at scale, and lack auditability. How Storage Actions Helps: Storage Actions enables teams to automate the preservation of validated training datasets using blob tags and immutability policies. Once a dataset is marked as clean and ready for training, Storage Actions can automatically: Lock the dataset using a time-based immutability policy Apply a tag to indicate it is a snapshot version This ensures that the dataset cannot be modified or deleted for the duration of the lock, and it is easily discoverable for future audits. Example in Practice: Let’s say an ML data pipeline tags a dataset with stage=clean after it passes validation and is ready for training. Storage Actions detects this tag and springs into action. It enforces a 1-year immutability policy, which means the dataset is locked and cannot be modified or deleted for the next 12 months. It also applies a tag snapshot=true, making it easy to locate and reference in future audits or investigations. The following conditions and operations define the task logic: IF: - Tags.Value[stage] equals 'clean' THEN: - SetBlobImmutabilityPolicy for 1-year: This adds a write once, read many (WORM) immutability policy on the blob to prevent deletion or modification, ensuring compliance. - SetBlobTags with snapshot=true: This adds a blob index tag with name “snapshot” and value “true”. Whenever this task runs on its scheduled interval - such as daily or weekly, it detects if a blob has the tag stage = 'clean', it automatically initiates the configured operations. In this case, Storage Actions applies a SetBlobImmutabilityPolicy on the blob for one year and adds a snapshot=true tag for easy identification. This means that without any manual intervention: The blob is made immutable for 12 months, preventing any modifications or deletions during that period. A snapshot=true tag is applied, making it easy to locate and audit later. No scripts, manual tagging, or access restrictions are needed to enforce data integrity. This ensures that validated training datasets are preserved in a tamper-proof state, satisfying audit and compliance requirements. It also reduces operational overhead by automating what would otherwise be a complex and error-prone manual process. Scenario 3: Embedding Management in AI Workflows Business Problem: Modern AI systems, especially those using Retrieval-Augmented Generation (RAG), rely heavily on vector embeddings to represent and retrieve relevant context from large document stores. These embeddings are often generated in real time, chunked into small files, and stored in vector databases or blob storage. As usage scales, these systems generate millions of small embedding files, many of which become obsolete quickly due to frequent updates, re-indexing, or model version changes. This silent accumulation of stale embeddings leads to: Increased storage costs Slower retrieval performance Operational complexity in managing the timings Traditionally, teams write scripts to purge old embeddings based on timestamps, run scheduled jobs, and manually monitor usage. This approach is brittle and does not scale well. How Storage Actions Helps: Storage Actions enables customers to automate the management of embeddings using blob tags and metadata. With blobs being identified with tags and metadata such as embeddings=true, modelVersion=latest, customers can define conditions that automatically delete stale embeddings without writing custom scripts. Example in Practice: In production RAG systems, embeddings are frequently regenerated to reflect updated content, new model versions, or refined chunking strategies. For example, a customer support chatbot may re-index its knowledge base daily to ensure responses are grounded in the latest documentation. To avoid bloating storage with outdated vector embeddings, Storage Actions can automate cleanup with task conditions and operation such as: IF: - Tags.Value[embeddings] equals 'true' - AND NOT Tags.Value[version] equals ‘latest’ - AND creation time < 12 days ago THEN: - DeleteBlob: This deletes all blobs which match the IF condition criteria. Whenever this Storage Action runs on its scheduled interval - such as daily - it scans for blobs that have the tag embeddings = ‘true’ and is not the latest version with its age being more than 12 days old, it automatically initiates the configured operation. In this case, Storage Actions does a DeleteBlob operation on the blob. This means that without any manual intervention: The stale embeddings are deleted No scripts or scheduled jobs are needed to track. This ensures that only the most recent model’s embeddings are retained, keeping the vector store lean and performant. It also reduces storage costs by eliminating obsolete data and helps maintain retrieval accuracy by ensuring outdated embeddings do not interfere with current queries. Applying Storage Actions to Storage Accounts To apply any of the scenarios, customers create an assignment during the storage task resource creation. In the assignment creation flow, they select the appropriate role and configure filters and trigger details. For example, a compliance cleanup scenario might run across the entire storage account with a recurring schedule every seven days to remove non-compliant blobs. A cost optimization scenario could target a specific container using a blob prefix and run as a one-time task to archive older blobs. A bulk tag update scenario would typically apply to all blobs without filtering and use a recurring schedule to keep tags consistent. After setting start and end dates, specifying the export container, and enabling the task, clicking Add queues the action to run on the account. Learn More If you are interested in exploring Storage Actions further, there are several resources to help you get started and deepen your understanding: Documentation on Getting Started: https://learn.microsoft.com/en-us/azure/storage-actions/storage-tasks/storage-task-quickstart-portal Create a Storage Action from the Azure Portal: https://portal.azure.com/#create/Microsoft.StorageTask Azure Storage Actions pricing: https://azure.microsoft.com/en-us/pricing/details/storage-actions/#pricing Azure Blog about the GA announcement: https://azure.microsoft.com/en-us/blog/unlock-seamless-data-management-with-azure-storage-actions-now-generally-available/ Azure Skilling Video with a walkthrough of Storage Actions: https://www.youtube.com/watch?v=CNdMFhdiNo8 Have questions, feedback, or a scenario to share? Drop a comment below or reach out to us at storageactions@microsoft.com. We would love to hear how you are using Storage Actions and what scenarios you would like to see next!
