Troubleshooting Azure Storage Deletion: Immutability, WORM, and Version Retention
One of the most common questions Azure Storage administrators ask after enabling immutable storage is: "Why can't I delete my storage account anymore?" Azure Immutable Storage is designed to provide WORM (Write Once, Read Many) protection, ensuring that data cannot be modified or deleted during a defined retention period. This capability is essential for compliance, regulatory, legal, and audit requirements, but it can be surprising when administrators later attempt to remove storage resources and discover that Azure refuses to delete them. In some environments, immutable storage policies may not have been configured directly by Azure Storage administrators. Backup or data protection solutions can automatically enable and manage retention settings, and storage administrators may not always have visibility into these externally managed policies. This article explains: Why storage accounts become undeletable Why containers sometimes appear empty but still cannot be deleted The difference between Unlocked and Locked policies How ImmutableStorageWithVersioning changes deletion behavior What happens when version retention is reduced to one day A practical step-by-step process to remove protected resources Understanding Azure Storage Immutability Azure Blob Storage supports immutability through two primary mechanisms: Time-Based Retention Policies A retention policy protects data for a fixed number of days. During this period: read are allowed, deletes are blocked and overwrites are blocked. Examples: 7 days,30 days,180 days,7 years Legal Holds Legal holds have no expiration date. The data remains protected until the hold is manually removed. Organizations commonly use legal holds when: litigation is in progress, regulatory investigations are occurring and data preservation is legally required Where Can Immutability Be Applied? Azure supports immutability at two different scopes. Scope Description Container-level Policy protects all blobs within the container Version-level Individual blob versions receive protection Version-level protection requires: Blob Versioning enabled and ImmutableStorageWithVersioning enabled Important: A storage account containing immutable data may itself become protected from deletion. What Happens When You Try to Delete Resources? Immutability blocks deletion at multiple layers. Think of it as a dependency chain: Storage Account └─ Container └─ Blob └─ Blob Version Azure will not allow deletion of a parent resource while protected child resources still exist. Scenario 1: Deleting the Storage Account You attempt to delete the storage account. Azure returns: Failed to delete storage account 'accountname'. Error: 'accountname' is protected from deletion. This occurs when at least one container still contains: a locked immutability policy, a legal hold or Immutable blob versions Before the account can be deleted: every user container must be removed, every blob must be removed, every blob version must be removed and every snapshot must be removed What About System Containers? System containers do not block deletion.These are managed automatically by Azure. The containers that matter are user-created containers. Examples: $logs, $blobchangefeed and $web Scenario 2: Deleting a Container You select Delete Container and Azure returns: The storage account container is protected from deletion due to ImmutableStorageWithVersioning. Even when, the container appears empty and the latest blobs have been deleted. Deletion may still fail. Why? Previous blob versions still exist. This is one of the most common causes of confusion. The Azure Portal normally displays: Only Show Active Blobs As a result, previous versions remain hidden. The container looks empty, but Azure still sees historical versions, snapshots and soft-deleted objects Although the container may appear empty at first glance, it is not actually empty. When "Show active and Deleted blobs" is enabled, the container displays all blobs that are currently in either the active or deleted state. Therefore, deletion remains blocked. Deleting containers with immutability (WORM) policies Container deletion may fail if an Immutable blob storage policy is applied at the container or version level. These policies keep blobs in a non-modifiable, delete-protected state (WORM) for the duration of the defined retention interval. The behavior depends on the policy State: Unlocked: the policy can still be edited or removed. You must delete the immutability policy reference from the container before deleting it. Once removed, the container and its blobs can be deleted normally. Locked: the policy cannot be deleted or shortened. You must wait until the retention interval expires for all affected blobs. Until that time elapses, the blobs, the container, and the storage account cannot be deleted. Scenario 3: Deleting Individual Blobs The retention policy prevents deletion of the blob regardless of whether the Delete option appears available in the user interface. Any delete request is rejected by the Storage service until the retention period expires. This indicates that an active retention policy is protecting the blob. Until retention expires: Delete, overwrite and modification are blocked. Unlocked Policies Think of this state as a testing phase. Protection is active, but administrators retain management flexibility. You can, Increase or decrease the retention period, or delete the policy. Once the policy is removed, the data becomes deletable. This allows organizations to validate configurations before committing to compliance requirements. Locked Policies A locked policy represents a compliance commitment. After locking: the policy cannot be deleted, retention cannot be reduced or extended. Most importantly: No user, administrator, or Microsoft Support engineer can bypass or remove a locked retention policy before expiration. i.e If retention is configured for 180 days, deletion becomes possible only after those 180 days have passed. The Hidden Detail: ImmutableStorageWithVersioning A common misunderstanding is believing that removing an unlocked retention policy completely disables immutability. It does not. Once ImmutableStorageWithVersioning is enabled: The feature itself cannot be disabled. Even if every unlocked policy is deleted, Azure still requires all versions to be removed before containers and storage accounts can be deleted. Why Version Retention Matters Blob versioning introduces another important behavior. When you modify version retention and reduce it to a lower value, for example: Once the version retention period is reduced (for example, from 180 days to 1 day), Azure automatically updates or creates the corresponding Lifecycle Management rule responsible for cleaning up old blob versions. At this stage, administrators can manually delete any remaining base blobs, snapshots, or versions that are already eligible for deletion. For versions that still exist and exceed the newly configured retention threshold: No additional manual action is required, the Lifecycle Management rule will automatically identify them as eligible for deletion and all versions older than the configured retention period are processed during subsequent Lifecycle Management executions. Example, If version retention is changed from 180 days to 1 day: Versions created more than 1 day ago immediately become eligible for cleanup, the Lifecycle Management engine evaluates those versions during its next execution cycle and once the versions are removed, container deletion can proceed provided no other immutable objects remain. Important Consideration Lifecycle Management is an asynchronous service. Therefore, after reducing retention: The configuration change is applied immediately, older versions become eligible for deletion immediately and actual version removal may take 24-48 hours or longer, depending on the Lifecycle Management execution schedule. For this reason, it is normal to see containers remain undeletable for a period of time after retention settings are modified. Administrators should allow sufficient time for Lifecycle Management to complete cleanup of all versions that exceed the configured retention threshold before attempting to delete the container or storage account. Azure lifecycle rule created automatically after changing the versioning retention. Many administrators are unaware that this happens behind the scenes. Code View for automatic rule created The purpose is simple: Remove older blob versions once they exceed the configured age. Reducing the retention period does not instantly delete existing versions. Instead, it makes existing versions eligible for deletion by Lifecycle Management, which removes them asynchronously during future processing cycles. Common Misconception Many customers assume: "Only versions created after I changed the setting will be deleted." This is incorrect. Azure evaluates every version independently using its creation timestamp. If a version is already older than one day when the rule runs: It becomes eligible for deletion. There is no grandfathering mechanism. Another Important Detail: Deletion Is Not Immediate Lifecycle Management follows an asynchronous process. Typical behavior: Action Expected Time New rule created Immediate Rule evaluated Within 24–48 hours Version cleanup begins During next lifecycle execution Because of this delay, containers may remain undeletable for some time even after retention settings are updated. Recommended Cleanup Process When attempting to remove immutable storage resources: Step 1 — Inventory all protection mechanisms: Account-level immutability, Container-level policies, Blob-level policies, Legal holds. Step 2 — Determine policy state: Locked or Unlocked Step 3 — Remove unlocked policies. Step 4 — Clear legal holds. Step 5 — Review versioning and soft delete settings. Step 6 — Remove all blobs, snapshots, and versions. Step 7 — Allow Lifecycle Management to remove old versions. Step 8 — Delete containers. Step 9 — Delete the storage account. Important Note: The deletion order is always blobs → container → storage account. A parent resource cannot be removed while protected children remain, so if any blob is still under a Locked policy (or an active legal hold), the entire process is blocked until retention expires or the hold is cleared. Key Takeaways ✅ Immutable storage protects data against deletion and modification. ✅ Locked policies cannot be bypassed, shortened, or removed. ✅ Legal holds must be manually cleared. ✅ Containers that appear empty may still contain hidden versions. ✅ ImmutableStorageWithVersioning cannot be turned off once enabled. ✅ Blob version retention automatically creates a Lifecycle Management rule. ✅ Storage account deletion requires every protected object underneath it to be removed first. References https://learn.microsoft.com/azure/storage/blobs/immutable-storage-overview https://learn.microsoft.com/azure/storage/blobs/immutable-policy-configure-container-scope https://learn.microsoft.com/azure/storage/blobs/immutable-policy-configure-version-scope https://learn.microsoft.com/azure/storage/blobs/data-protection-overview https://learn.microsoft.com/azure/storage/blobs/lifecycle-management-overviewAzure Retirement Livestream - Please register! Session 2 - Tracking ID: XTKT-BW8
Join our upcoming live webcast for a transparent discussion about this upcoming Azure retirement — led by our engineering teams. General Purpose v1 (GPv1) Storage Accounts Tracking ID: XTKT-BW8 | Retirement Date: 13 October 2026 Same content presented in both sessions — pick the one that works best for your timezone! What to expect 📚 Understand What will happen, the timelines for the change, and how you can manage it 💬 Ask Live Q&A with our engineering experts throughout the session 🛠 Learn How to manage the change smoothly Choose your session Same content presented at both times — pick the one that works best for your timezone: Session 1 14:30 UTC Thursday, 25 June 2026 Register now → Session 2 04:30 UTC Friday, 26 June 2026 Register now → 8:30 AM US Pacific (PDT) 11:30 AM US Eastern (EDT) 4:30 PM London (BST) 12:30 AM +1 Beijing (CST) 3:30 AM +1 Sydney (AEDT) 5:30 AM +1 Auckland (NZDT) 8:30 PM -1 US Pacific (PDT) 11:30 PM US Eastern (EDT) 4:30 AM London (BST) 12:30 PM Beijing (CST) 3:30 PM Sydney (AEDT) 5:30 PM Auckland (NZDT) Our engineering leaders George Trossell Senior Product Manager Azure Networking LinkedIn ↗ ⚠️ Prepare before the livestream Read the Post Incident Review (PIR) ahead of time so you can ask any follow up questions during the live Q&A Helpful resources 🔔 Azure Service Health Alerts Get alerts for relevant incidents by setting up notifications via email, SMS, or webhook 🎥 Past Retrospective Recordings Watch recordings of previous retrospective livestreams 📄 Azure Post Incident Reviews Learn more about PIRs and the retrospective programAzure Retirement Livestream - Please register! Session 1- Tracking ID: XTKT-BW8
Join our upcoming live webcast for a transparent discussion about this upcoming Azure retirement — led by our engineering teams. General Purpose v1 (GPv1) Storage Accounts Tracking ID: XTKT-BW8 | Retirement Date: 13 October 2026 Same content presented in both sessions — pick the one that works best for your timezone! What to expect 📚 Understand What will happen, the timelines for the change, and how you can manage it 💬 Ask Live Q&A with our engineering experts throughout the session 🛠 Learn How to manage the change smoothly Choose your session Same content presented at both times — pick the one that works best for your timezone: Session 1 14:30 UTC Thursday, 25 June 2026 Register now → Session 2 04:30 UTC Friday, 26 June 2026 Register now → 8:30 AM US Pacific (PDT) 11:30 AM US Eastern (EDT) 4:30 PM London (BST) 12:30 AM +1 Beijing (CST) 3:30 AM +1 Sydney (AEDT) 5:30 AM +1 Auckland (NZDT) 8:30 PM -1 US Pacific (PDT) 11:30 PM US Eastern (EDT) 4:30 AM London (BST) 12:30 PM Beijing (CST) 3:30 PM Sydney (AEDT) 5:30 PM Auckland (NZDT) Our engineering leaders George Trossell Senior Product Manager Azure Networking LinkedIn ↗ ⚠️ Prepare before the livestream Read the Post Incident Review (PIR) ahead of time so you can ask any follow up questions during the live Q&A Helpful resources 🔔 Azure Service Health Alerts Get alerts for relevant incidents by setting up notifications via email, SMS, or webhook 🎥 Past Retrospective Recordings Watch recordings of previous retrospective livestreams 📄 Azure Post Incident Reviews Learn more about PIRs and the retrospective programHow to get blob Total Blob Count and Total Capacity with Blob Inventory
Approach This article presents how to take advantage of the Blob Inventory service to get the total blob count and the total capacity per storage account, per container or per directory. I will present the steps to create the blob inventory rule and how to get the needed information without having to process the blob inventory rule results just by using the prefix match field. Additional support documentation it is presented at the end of the article. Introduction to the Blob Inventory Service Azure Storage blob inventory provides a list of the containers, blobs, blob versions, and snapshots in your storage account, along with their associated properties. It generates an output report in either comma-separated values (csv) or Apache Parquet format on a daily or weekly basis. You can use the report to audit retention, legal hold or encryption status of your storage account contents, or you can use it to understand the total data size, age, tier distribution, or other attributes of your data. Please find here our documentation about the blob inventory service. On this article, I will focus on using this service to get the blob count and the capacity. Steps to enable inventory report Please see below how to define a blob inventory rule to get the intended information, using the Azure Portal: Sign in to the Azure portal to get started. Locate your storage account and display the account overview. Under Data management, select Blob inventory. Select Add your first inventory rule, if you do not have any rule defined, or select Add a rule, in case that you already have at least one rule defined. Add a new inventory rule by filling in the following fields: Rule name: The name of your blob inventory rule. Container: Container to store the result of the blob inventory rule execution. Object type to inventory: Select blob Blob types: Blob Storage: Select all (Block blobs, Page blobs, Append blobs). Data Lake Storage: Select all (Block blobs, Append blobs). Subtypes: Blob Storage: Select all (Include blob versions, Include snapshots, Include deleted blobs). Data Lake Storage: Select all (Include snapshots, Include deleted blobs). Blob inventory fields: Please find here all custom schema fields supported for blob inventory. In this scenario, we need to select at least the following fields: Blob Storage: Name, Creation-Time, ETag, Content-Length, Snapshot, VersionId, IsCurrentVersion, Deleted, RemainingRetentionDays. Data Lake Storage: Name, Creation-Time, ETag, Content-Length, Snapshot, DeletionId, Deleted, DeletedTime, RemainingRetentionDays. Inventory frequency: A blob inventory run is automatically scheduled every day when daily is chosen. Selecting weekly schedule will only trigger the inventory run on Sundays. A daily execution will return results faster. Export format: The export format. Could be a csv file or a parquet file. Prefix match: Filter blobs by name or first letters. To find items in a specific container, enter the name of the container followed by a forward slash, then the blob name or first letters. For example, to show all blobs starting with “a”, type: “myContainer/a”. Here is the place to add the path where to start collecting the blob information. The step 5.9 presented above (the prefix match field) it is the main point of this article. Considering that we have a Storage Account with a container named work, and a directory named items, inside the container named work. Please see below how to configure the prefix match field to get the needed result: Leave it empty to get the information at the storage account level. Add the container name in the prefix match field to get the information at the container level. Put prefix match = work/ Add the directory path in the prefix match field to get the information at the directory level. Put prefix match = work/items/ The blob inventory execution will generate a file named <ruleName>-manifest.json, please see more information about this file in the support documentation section. This file captures the rule definition provided by the user and the path to the inventory for that rule, and the information that we want without having to process the blob inventory rule files. { "destinationContainer" : "inventory-destination-container", "endpoint" : "https://testaccount.blob.core.windows.net", "files" : [ { "blob" : "2021/05/26/13-25-36/Rule_1/Rule_1.csv", "size" : 12710092 } ], "inventoryCompletionTime" : "2021-05-26T13:35:56Z", "inventoryStartTime" : "2021-05-26T13:25:36Z", "ruleDefinition" : { "filters" : { "blobTypes" : [ "blockBlob" ], "includeBlobVersions" : false, "includeSnapshots" : false, "prefixMatch" : [ "penner-test-container-100003" ] }, "format" : "csv", "objectType" : "blob", "schedule" : "daily", "schemaFields" : [ "Name", "Creation-Time", "BlobType", "Content-Length", "LastAccessTime", "Last-Modified", "Metadata", "AccessTier" ] }, "ruleName" : "Rule_1", "status" : "Succeeded", "summary" : { "objectCount" : 110000, "totalObjectSize" : 23789775 }, "version" : "1.0" } The objectCount value is the total blob count, and the totalObjectSize is the total capacity in bytes. Special notes: A rule needs to be defined for each path (container or directory) to get the total blob count and the total capacity. The blob inventory rule generates a CSV or Apache Parquet formatted file(s). These files should be deleted if the blob inventory rule is only to get the information presented on this article. Support Documentation Topic Some highlights Enable Azure Storage blob inventory reports The steps to enable inventory report. Inventory run If you configure a rule to run daily, then it will be scheduled to run every day. If you configure a rule to run weekly, then it will be scheduled to run each week on Sunday UTC time. The time taken to generate an inventory report depends on various factors and the maximum amount of time that an inventory run can complete before it fails is six days. Inventory output Each inventory rule generates a set of files in the specified inventory destination container for that rule. The inventory output is generated under the following path: https://<accountName>.blob.core.windows.net/<inventory-destination-container>/YYYY/MM/DD/HH-MM-SS/<ruleName where: accountName is your Azure Blob Storage account name. inventory-destination-container is the destination container you specified in the inventory rule. YYYY/MM/DD/HH-MM-SS is the time when the inventory began to run. ruleName is the inventory rule name. Inventory files Each inventory run for a rule generates the following files: Inventory file: An inventory run for a rule generates a CSV or Apache Parquet formatted file. Each such file contains matched objects and their metadata. Checksum file: A checksum file contains the MD5 checksum of the contents of manifest.json file. The name of the checksum file is <ruleName>-manifest.checksum. Generation of the checksum file marks the completion of an inventory rule run Manifest file: A manifest.json file contains the details of the inventory file(s) generated for that rule. The name of the file is <ruleName>-manifest.json. This file also captures the rule definition provided by the user and the path to the inventory for that rule. Pricing and billing Pricing for inventory is based on the number of blobs and containers that are scanned during the billing period. Known Issues and Limitations This section describes limitations and known issues of the Azure Storage blob inventory feature. Disclaimer These steps are provided for the purpose of illustration only. These steps and any related information are provided "as is" without warranty of any kind, either expressed or implied, including but not limited to the implied warranties of merchantability and/or fitness for a particular purpose. We grant You a nonexclusive, royalty-free right to use and modify the Steps and to reproduce and distribute the steps, provided that. You agree: to not use Our name, logo, or trademarks to market Your software product in which the steps are embedded; to include a valid copyright notice on Your software product in which the steps are embedded; and to indemnify, hold harmless, and defend Us and Our suppliers from and against any claims or lawsuits, including attorneys’ fees, that arise or result from the use or distribution of steps.
[Design Pattern] Handling race conditions and state in serverless data pipelines
Hello community, I recently faced a tricky data engineering challenge involving a lot of Parquet files (about 2 million records) that needed to be ingested, transformed, and split into different entities. The hard part wasn't the volume, but the logic. We needed to generate globally unique, sequential IDs for specific columns while keeping the execution time under two hours. We were restricted to using only Azure Functions, ADF, and Storage. This created a conflict: we needed parallel processing to meet the time limit, but parallel processing usually breaks sequential ID generation due to race conditions on the counters. I documented the three architecture patterns we tested to solve this: Sequential processing with ADF (Safe, but failed the 2-hour time limit). 2. Parallel processing with external locking/e-tags on Table Storage (Too complex and we still hit issues with inserts). 3. A "Fan-Out/Fan-In" pattern using Azure Durable Functions and Durable Entities. We ended up going with Durable Entities. Since they act as stateful actors, they allowed us to handle the ID counter state sequentially in memory while the heavy lifting (transformation) ran in parallel. It solved the race condition issue without killing performance. I wrote a detailed breakdown of the logic and trade-offs here if anyone is interested in the implementation details: https://medium.com/@yahiachames/data-ingestion-pipeline-a-data-engineers-dilemma-and-azure-solutions-7c4b36f11351 I am curious if others have used Durable Entities for this kind of ETL work, or if you usually rely on an external database sequence to handle ID generation in serverless setups? Thanks, Chameseddine
Deriving expiry days and remaining retention days for blobs through blob inventory
In managing data within Azure blob storage accounts and Azure data lake gen 2 storage accounts, organizations often encounter scenarios where blobs have been deleted but remain in a soft-deleted state. To calculate the remaining retention days for all such blobs across an entire storage account can be a critical requirement for customers seeking to optimize data management and ensure compliance with retention policies. Additionally, certain blobs may have an expiry time set, scheduling their deletion for a future date. To facilitate the identification and monitoring of these blobs and their respective expiry times, a custom query has been written to efficiently list and calculate expiry information, enabling users to proactively manage their storage resources. The expiry time for Azure blobs is set using the Set Blob Expiry operation. This feature is present in only Hierarchical namespace enabled storage accounts. We can set the expiry with below steps: i) Azure Storage action- About Azure Storage Actions - Azure Storage Actions | Microsoft Learn Storage action can be used to set blob expiry, share a high-level snippet for the operation below ii) REST API- Set Blob Expiry (REST API) - Azure Storage | Microsoft Learn to set the expiry time for your blobs. This ensures that each blob has a defined lifecycle and will be deleted after the specified period. In this blog, it is a step-by-step process of listing the expiry time and retention of the blobs using Blob Inventory report and then parsing it using Synapse. 1. Set blob inventory rule Get CSV file from blob inventory run. Go to the container where inventory reports are getting stored. Navigate to the recent date folder and get url of Blob Inventory csv life. Sharing the below snippet for reference: 2. Create an Azure Synapse workspace Next, create an Azure Synapse workspace where you will execute a SQL query to report the inventory results. Create the SQL query: After you create your Azure Synapse workspace, do the following steps. Navigate to https://web.azuresynapse.net. Select the Develop tab on the left edge. Select the large plus sign (+) to add an item. Select SQL script. 3. Use the sample query below to get the expiry time and remaining retention days of blob respectively select LEFT([Name], CHARINDEX('/', [Name]) - 1) AS Container, RIGHT([Name], LEN([Name])- CHARINDEX('/',[Name])) AS Blob, [Expiry-time] from OPENROWSET( bulk '<URL to your inventory CSV file>', format='csv', parser_version='2.0', header_row=true ) as Source For blobs which got deleted directly, you can calculate the remaining retention days since the data is present in soft deleted state and will be deleted permanently after the retention days complete. select LEFT([Name], CHARINDEX('/', [Name]) - 1) AS Container, RIGHT([Name], LEN([Name])- CHARINDEX('/',[Name])) AS Blob, [Expiry-time], RemainingRetentionDays from OPENROWSET( bulk '<URL to your inventory CSV file>', format='csv', parser_version='2.0', header_row=true ) as Source In the above snippet, the Null value represents that the blob is not deleted and no expiry time is set on the blob yet. Please Note: Calculating blob expiry from blob inventory is one way, customer can explore other options such as Powershell and Azure CLI to achieve the same. Reference links:- Set Blob Expiry (REST API) - Azure Storage | Microsoft Learn Create a storage task - Azure Storage Actions | Microsoft Learn Azure Storage blob inventory | Microsoft Learn Calculate blob count and size using Azure Storage inventory | Microsoft Learn
Exclude Prefix in Azure Storage Action: Smarter Blob Management
Azure Storage Actions is a powerful platform for automating data management tasks across Blob and Data Lake Storage. Among its many features, Exclude Prefix stands out as a subtle yet critical capability that helps fine-tune task assignments. What Is the "Exclude Prefix" Feature? The Exclude Prefix option allows users to omit specific blobs or folders from being targeted by Azure Storage Actions. This is particularly useful when applying actions such as: Moving blobs to a cooler tier Deleting blobs Rehydrating archived blobs Triggering workflows based on blob changes For example, if you're running a task to archive blobs older than 30 days, but you want to exclude logs or config files, you can define a prefix like logs/ or config/ in the exclusion list. How to Use It — Example Scenario: In the following example, blobs across the entire storage account were deleted based on a condition: if a blob’s access tier was set to Hot, it was deleted except for those blobs or paths explicitly listed under the Exclude blob prefixes property. Create a Task: - Navigate to the Azure portal and search for Storage tasks. Then, under Services, click on Storage tasks – Azure Storage Actions On the Azure Storage Actions | Storage Tasks page, click Create to begin configuring a new task. Complete all the required fields, then click Next to proceed to the Conditions page. To configure blob deletion, add the following conditions on the Conditions page. Add the Assignment :- Click Add assignment in the Select scope section, choose your subscription and storage account, then provide a name for the assignment. In the Role assignment section, select Storage Blob Data Owner from the Role drop-down list to assign this role to the system-assigned managed identity of the storage task. In the Filter objects section, specify the Exclude Blob Prefix filter if you want to exclude specific blobs or folders from the task. In the example specified above, blobs will be deleted—except for those under the path “excludefiles” listed in the Exclude blobprefixes property. In the Trigger details section, choose the runs of the task and then select the container where you'd like to store the execution reports. Select Add. In the Tags tab, select Next and in the Review + Create tab, select Review + create. When the task is deployed, your deployment is complete page appears and select Go to resource to open the Overview page of the storage task. Enable the Task Assignment: - In the Trigger details section, we have a Enable task assignment checkbox which is checked by default. If the Enable task assignments checkbox is unchecked, you can still enable assignments manually from the Assignments page. To do this, go to Assignments, select the relevant assignment, and then click Enable. The task assignment is queued to run and will run at the specified time. Monitoring the runs:- After the task completes running, you can view the results of the run. With the Assignments page still open, select View task runs. Select the View report link to download a report. You can also view these comma-separated reports in the container that you specified when you configured the assignment. Conclusion: The Exclude Prefix feature in Azure Storage Actions provides enhanced control and flexibility when managing blob data at scale. By allowing you to exclude specific prefixes from actions like delete or tier changes, it helps you safeguard critical data, reduce mistakes, and fine-tune automation workflows. This targeted approach not only improves operational efficiency but also supports more granular data in Azure Blob Storage. Note:- Azure Storage Actions are generally available in the following public regions: https://learn.microsoft.com/en-us/azure/storage-actions/overview#supported-regions We can also exclude certain blobs using the “Not”operator when building task conditions. Blobs may be excluded based on specific blob or container attributes from the task conditions side as well—not just through task assignments. In the screenshot below, we are using the Not operator (!) to exclude blobs where the blob name is equal to "Test". Please refer: https://learn.microsoft.com/en-us/azure/storage-actions/storage-tasks/storage-task-conditions#multiple-clauses-in-a-condition. Reference Links:- About Azure Storage Actions - Azure Storage Actions | Microsoft Learn Storage task best practices - Azure Storage Actions | Microsoft Learn Known issues and limitations with storage tasks - Azure Storage Actions | Microsoft LearnConverting Page or Append Blobs to Block Blobs with ADF
In certain scenarios, a storage account may contain a significant number of page blobs classified under the hot access tier that are infrequently accessed or retained solely for backup purposes. To optimise costs, it is desirable to transition these page blobs to the archive tier. However, as indicated in the following documentation - https://learn.microsoft.com/en-us/azure/storage/blobs/access-tiers-overview the ability to set the access tier is only available for block blobs; this functionality is not supported for append or page blobs. The Azure blob storage connector in Azure data factory is capable of copying blobs from block, append, or page blobs and copying data to only block blobs. https://learn.microsoft.com/en-us/azure/data-factory/connector-azure-blob-storage?tabs=data-factory#supported-capabilities Note: No extra configuration is required to set the blob type on the destination. By default, the ADF copy activity creates blobs as Block Blobs. In this blog, we will understand how to make use of Azure Data Factory to copy the page blobs to block blobs. Please note that this is applicable to append blobs as well. Let’s take a look at the steps ahead Step 1: Creating ADF instance Create an Azure data factory resource in the Azure portal referring to the following document - https://learn.microsoft.com/en-us/azure/data-factory/quickstart-create-data-factory After creation, click on "Launch Studio" as shown below Step 2: Creating datasets Create two datasets by navigating to Author -> Datasets -> New dataset. These datasets are used in source and sink for the ADF copy activity Select "Azure blob storage" -> click on continue -> select "binary" and continue Step 3: Creating Linked service Create a new linked service and provide the storage account name which contains page blobs Provide the file path where the page blobs are located. You would also need to create another dataset for destination. Repeat the steps from 3 to 6 to create another destination dataset to copy the blobs to the storage account as block blobs. Note: You can use same or different storage account for the destination dataset. Set it as per your requirements. Step 4: Configuring a Copy data pipeline Once the two datasets are created, now create a new pipeline and under "Move and Transform" section, drag and drop the "Copy data" activity as shown below. Under the Source and Sink sections from the drop down, select the source and destination datasets respectively which were created in the previous steps. Select the “Recursively” option and publish the changes. Source: Sink: Note: You can configure the filters and copy behaviour as per your requirements. Step 5: Debugging and validating Now as the configuration is completed, click on "Debug". If the pipeline activity ran successfully, you should be able to see "succeeded" status in the output section as below. Verify the blob type of the blobs in the destination storage account and it should show as block blob and access tier as Hot. After converting the blobs to block blobs, several methods are available to change their access tier to archive. These include implementing a blob lifecycle management policy, utilizing storage actions, or by using Az CLI or PowerShell scripts. Conclusion Utilising ADF enables the conversion of page or append blobs to block blobs, after which any standard method such as LCM policy or storage actions may be used to change the access tier to archive. This strategy offers a more streamlined and efficient solution compared to developing custom code or scripts. Reference links: https://learn.microsoft.com/en-us/azure/storage/blobs/access-tiers-overview https://learn.microsoft.com/en-us/azure/data-factory/connector-azure-blob-storage?tabs=data-factory#supported-capabilities https://learn.microsoft.com/en-us/azure/data-factory/copy-activity-overview https://learn.microsoft.com/en-us/azure/storage-actions/storage-tasks/storage-task-quickstart-portal https://learn.microsoft.com/en-us/azure/storage/blobs/lifecycle-management-overview https://learn.microsoft.com/en-us/azure/storage/blobs/archive-blob?tabs=azure-powershell#bulk-archive
Rehydrating Archived Blobs via Storage Task Actions
Azure Storage Actions is a fully managed platform designed to automate data management tasks for Azure Blob Storage and Azure Data Lake Storage. You can use it to perform common data operations on millions of objects across multiple storage accounts without provisioning extra compute capacity and without requiring you to write code. Storage task actions can be used to rehydrate the archived blobs in any tier as required. Please note there is no option to set the rehydration priority and is defaulted to Standard one as of now. Note :- Azure Storage Actions are generally available in the following public regions: https://learn.microsoft.com/en-us/azure/storage-actions/overview#supported-regions Azure Storage Actions is currently in PREVIEW in following reions. Please refer: https://learn.microsoft.com/en-us/azure/storage-actions/overview#regions-supported-at-the-preview-level Below are the steps to achieve the rehydration :- Create a Task :- In the Azure portal, search for Storage tasks. Under Services, select Storage tasks - Azure Storage Actions. On the Azure Storage Actions | Storage Tasks page, select Create Fill in all the required details and click on Next to open the Conditions page. Add the conditions as below if you want to rehydrate to Cool tier. Add the Assignment :- Select Add assignment. In the Select scope section, select your subscription and storage account and name the assignment. In the Role assignment section, in the Role drop-down list, select the Storage Blob Data Owner to assign that role to the system-assigned managed identity of the storage task. In the Filter objects section, specify the filter if you want this to run on some specific objects or the whole storage account. In the Trigger details section, choose the runs of the task and then select the container where you'd like to store the execution reports. Select Add. In the Tags tab, select Next. In the Review + Create tab, select Review + create. When the task is deployed, Your deployment is complete page appears. Select Go to resource to open the Overview page of the storage task. Enable the Task Assignment :- Storage task assignments are disabled by default. Enable assignments from the Assignments page. Select Assignments, select the assignment, and then select Enable. The task assignment is queued to run and will run at the specified time. Monitoring the runs :- After the task completes running, you can view the results of the run. With the Assignments page still open, select View task runs. Select the View report link to download a report. You can also view these comma-separated reports in the container that you specified when you configured the assignment. Reference Links :- About Azure Storage Actions - Azure Storage Actions | Microsoft Learn Storage task best practices - Azure Storage Actions | Microsoft Learn Known issues and limitations with storage tasks - Azure Storage Actions | Microsoft Learn
Lifecycle Management of Blobs (Deletion) using Automation Tasks
Background: We often encounter scenarios where we need to delete blobs that have been idle in a storage account for an extended period. For a small number of blobs, deletion can be handled easily using the Azure Portal, Storage Explorer, or inline scripts such as PowerShell or Azure CLI. However, in most cases, we deal with a large volume of blobs, making manual deletion impractical. In such situations, it's essential to leverage automation tools to streamline the deletion process. One effective option is using Automation Tasks, which can help schedule and manage blob deletions efficiently. Note: Behind the scenes, an automation task is actually a logic app resource that runs a workflow. So, the Consumption pricing model of logic-app applies to automation tasks. Scenario’s where “Automation Tasks” are helpful: You have a requirement to automate deletion of blobs which are older than a specific time, in days, weeks or months. You don’t want to put in much manual effort rather have simple UI based steps to achieve your goal You have System containers, and you want to action on it. We have “LCM (Life Cycle management)” which too can be leveraged by users to automation deletion of older blobs; however LCM cannot be used to delete blobs from System containers. You have to work on page blobs. Setup “Automation Tasks”: Let’s walk through on how to achieve our goal. Navigate to the desired storage account and scroll down to the “Automation” section and select the “Tasks” blade and then click on “Add Task” from the top panel or bottom panel (highlighted in image). On the next page click the “Select” (highlighted image) The new page which opens up should look as below, however there isn’t anything we are doing. So let’s just click on the “Next : Configure” (highlighted in image) and move to the next screen. The new page opens needs to be filled as per your requirement. I have added a sample. You can also use it on your own containers as well. 'sample' is a folder inside container '$web' The “Expiration Age” field means that Blobs older than these number of days needs be deleted. In above screenshot, blobs older than 180 days would be deleted. Similarly, we can configure values in weeks or months as well. Once we are through with the steps proceed with creation of the task. Once task is created it looks as below: You can click on the “View Run” to see run history. In-case you want to modify the task, click on your tasks name. For example in above screenshot I can modify by clicking “mytask” link and re-configure the task. Now this isn’t sufficient. We will update some of the steps which were used to create the Logic-app. Hence we would need to edit some steps and save those before re-running the app. a) Go the logic app and navigate to the “Logic App Designer” blade b) Now click on the “+” sign as shown below and “Add an Action” c) Once the new page opens up, search for “List Blobs (v2)” and select it d) Choose the “Enter custom value” and enter your storage account name e) The values would like as shown below f) Now let's navigate to the “For Each” condition g) We need to delete the “Delete blob” too and replace with “Delete blob (V2)” h) The “Delete Blob (V2)” looks like as below i) With all steps ready, lets save the logic app and click on “Run”. You should observe the run passing successfully. Impact due to Firewall: The above steps for works when your storage account is configured for public access. However, when firewall is enabled, you would need to provide the necessary permissions, else you are going to encounter 403 "Authorization Failure" errors. There would be no issue to create the task, but you will see failures when you check the runs. Example: To overcome this limitation, you need to navigate to your logic app and generate a managed identity for the app and provide the identity “Storage Blob Data Contributor” role. Step1. Enable Managed Identity: In Azure Portal, go to your Logic App resource. Under Settings, select Identity. In the Identity pane, under System assigned, select On and Save. This step registers the system-assigned identity with Microsoft Entra ID, represented by an object ID. Step2. Assign Necessary Role: Open the Azure Storage Account in Azure Portal. Select Access control (IAM) > Add > Add role assignment. Assign a role like 'Storage Blob Data Contributor', which includes write access for blobs in an Azure Storage container, to the managed identity. Under Assign access to, select Managed identity > Add members, and choose your Logic App's identity Save and refresh and you see the new role configured to your storage account Remember that, if the Storage Account and logic app are in different region you should add another step in the firewall of storage account. You need to whitelist the logic app instance in “Resource instances” list as shown below: Conclusion: The multiple ways to action on blobs are provided for your convenience. Depending on your requirement, feasibility & other factors like comfortability with the feature or pricing too would certainly influence your decisions. However, in-case you want to action upon System containers like $logs or $web, “Automation Tasks” are one of the most helpful feature which you can use and achieve your goal. Note: At the time of writing this blog this feature is still in preview. So ensure to check if there are any limitations which might impact you before implementing it in your Production environment. References: Create automation tasks to manage and monitor Azure resources - Azure Logic Apps | Microsoft Learn Optimize costs by automatically managing the data lifecycle - Azure Blob Storage | Microsoft Learn
