SSL/TLS connection issue troubleshooting guide
You may experience exceptions or errors when establishing TLS connections with Azure services. Exceptions are vary dramatically depending on the client and server types. A typical ones such as "Could not create SSL/TLS secure channel." "SSL Handshake Failed", etc. In this article we will discuss common causes of TLS related issue and troubleshooting steps.
Azure Data Factory trigger is not initiated when uploading a file using Java SDK
Uploading file using Java SDK class DataLakeFileClient does not initiate an ADF trigger despite configuring the ADF trigger correctly to be initiated once a new file is created. This is happening only when the trigger is configured to ignore 0 byte blobs.
Set Up Endpoint DLP Evidence Collection on your Azure Blob Storage
Endpoint Data Loss Prevention (Endpoint DLP) is part of the Microsoft Purview Data Loss Prevention (DLP) suite of features you can use to discover and protect sensitive items across Microsoft 365 services. Microsoft Endpoint DLP allows you to detect and protect sensitive content across onboarded Windows 10, Windows 11 and macOS devices. Learn more about all of Microsoft's DLP offerings. Before you start setting up the storage, you should review Get started with collecting files that match data loss prevention policies from devices | Microsoft Learn to understand the licensing, permissions, device onboarding and your requirements. Prerequisites Before you begin, ensure the following prerequisites are met: You have an active Azure subscription. You have the necessary permissions to create and configure resources in Azure. You have setup endpoint Data Loss Prevention policy on your devices Configure the Azure Blob Storage You can follow these steps to create an Azure Blob Storage using the Azure portal. For other methods refer to Create a storage account - Azure Storage | Microsoft Learn Sign in to the Azure Storage Accounts with your account credentials. Click on + Create On the Basics tab, provide the essential information for your storage account. After you complete the Basics tab, you can choose to further customize your new storage account, or you accept the default options and proceed. Learn more about azure storage account properties Once you have provided all the information click on the Networking tab. In network access, select Enable public access from all networks while creating the storage account. Click on Review + create to validate the settings. Once the validation passes, click on Create to create the storage Wait for deployment of the resource to be completed and then click on Go to resource. Once the newly created Blob Storage is opened, on the left panel click on Data Storage -> Containers Click on + Containers. Provide the name and other details and then click on Create Once your container is successfully created, click on it. Assign relevant permissions to the Azure Blob Storage Once the container is created, using Microsoft Entra authorization, you must configure two sets of permissions (role groups) on it: One for the administrators and investigators so they can view and manage evidence One for users who need to upload items to Azure from their devices Best practice is to enforce least privilege for all users, regardless of role. By enforcing least privilege, you ensure that user permissions are limited to only those permissions necessary for their role. We will use portal to create these custom roles. Learn more about custom roles in Azure RBAC Open the container and in the left panel click on Access Control (IAM) Click on the Roles tab. It will open a list of all available roles. Open context menu of Owner role using ellipsis button (…) and click on Clone. Now you can create a custom role. Click on Start from scratch. We have to create two new custom roles. Based on the role you are creating enter basic details like name and description and then click on JSON tab. JSON tab gives you the details of the custom role including the permissions added to that role. For owner role JSON looks like this: Now edit these permissions and replace them with permissions required based on the role: Investigator Role: Copy the permissions available at Permissions on Azure blob for administrators and investigators and paste it in the JSON section. User Role: Copy the permissions available at Permissions on Azure blob for usersand paste it in the JSON section. Once you have created these two new roles, we will assign these roles to relevant users. Click on Role Assignments tab, then on Add + and on Add role assignment. Search for the role and click on it. Then click on Members tab Click on + Select Members. Add the users or user groups you want to add for that role and click on Select Investigator role – Assign this role to users who are administrators and investigators so they can view and manage evidence User role – Assign this role to users who will be under the scope of the DLP policy and from whose devices items will be uploaded to the storage Once you have added the users click on Review+Assign to save the changes. Now we can add this storage to DLP policy. For more information on configuring the Azure Blob Storage access, refer to these articles: How to authorize access to blob data in the Azure portal Assign share-level permissions. Configure storage in your DLP policy Once you have configured the required permissions on the Azure Blob Storage, we will add the storage to DLP endpoint settings. Learn more about configuring DLP policy Open the storage you want to use. In left panel click on Data Storage -> Containers. Then select the container you want to add to DLP settings. Click on the Context Menu (… button) and then Container Properties. Copy the URL Open the Data Loss Prevention Settings. Click on Endpoint Settings and then on Setup evidence collection for file activities on devices. Select Customer Managed Storage option and then click on Add Storage Give the storage name and copy the container URL we copied. Then click on Save. Storage will be added to the list. Storage will be added to the list for use in the policy configuration. You can add up to 10 URLs Now open the DLP endpoint policy configuration for which you want to collect the evidence. Configure your policy using these settings: Make sure that Devices is selected in the location. In Incident reports, toggle Send an alert to admins when a rule match occurs to On. In Incident reports, select Collect original file as evidence for all selected file activities on Endpoint. Select the storage account you want to collect the evidence in for that rule using the dropdown menu. The dropdown menu shows the list of storages configured in the endpoint DLP settings. Select the activities for which you want to copy matched items to Azure storage Save the changes Please reach out to the support team if you face any issues. We hope this guide is helpful and we look forward to your feedback. Thank you, Microsoft Purview Data Loss Prevention TeamHow to calculate Container Level Statistics in Azure Blob Storage with Azure Databricks
This article describes how to get container level stats in Azure Blob Storage, and how to work with the information provided by blob inventory. The approach presented here uses Azure Databricks and is most suited to be used in storage accounts with a huge amount of data, in order of Terabytes (TB) or more.Step-by-Step Guide: Setting up Custom Domain for Azure Storage Account with HTTPS Only Enabled
If you are using Azure Storage to host your website, you might want to enable HTTPS Only to ensure secure communication between the client and the server. However, setting up a custom domain with HTTPS Only enabled can be a bit tricky. In this blog, we will guide you through the step-by-step process of setting up a custom domain for your Azure Storage account with HTTPS Only enabled. Firstly, I would like to briefly explain that currently, configuring a custom domain directly on Azure Storage is only supported for HTTP. If HTTP access is allowed, you can follow this official documentation for configuration: https://learn.microsoft.com/en-us/azure/storage/blobs/storage-custom-domain-name?tabs=azure-portal#map-a-custom-domain-with-only-http-enabled Otherwise, if we need to use a custom domain with HTTPS Only enabled, we need to leverage other resources on Azure to do request forwarding. We will explore three options for setting up a custom domain in this scenario. These three options have their own pros and cons, and a brief comparison will also be provided at the end of the blog. You can choose to use either of them according to your specific situation. Azure CDN Application Gateway Azure Front Door Please note that if you use either request forwarder, you no longer need to manually add the custom domain to storage. If you add the custom domain in this case, you may get "cannot verify" error. Option 1: Using Azure CDN Official Document Reference: https://learn.microsoft.com/en-us/azure/storage/blobs/storage-custom-domain-name?tabs=azure-portal#using-azure-cdn If you have your own DNS provider already, you can ignore the first four steps and start with Step 5. [Step 1] First of all, you should get a domain. On Azure, you can deploy “App Service Domain” which will create a DNS zone for you automatically. [Step 2] You need to purchase a certificate and create a key vault (if you have any existing KVs, you can use an existing one). After the Key Vault is created, you need to go to the Access configuration of the KV and update it to “Vault access policy”. [Step 3] You need to go to the certificate that you just created. Then, you click the Certificate Configuration and have three boxes checked one by one. a. You select the key vault you just created to link with the certificate. b. You verify the domain you created earlier. This step may take a few minutes to take effect after checking for domain verification with certificate issuer. [Step 4] Now, you can add a CNAME record in your DNS zone, like below. [Step 5] You can create a new CDN profile on Azure if you don’t have one already. This step is within your storage account, where you can navigate to "Front door and CDN" in the left-hand menu. [Step 6] Then, you should be able to add a custom domain in your Endpoint of CDN. [Step 7] You need to enable the HTTPS feature for custom domain, you can click in the custom domain you just added in Step 6 and enable the HTTPS. (This process may take up to a few hours to finish.) To sum up, the traffic is going from external hostname (CNAME record name you added in Step 4: in my example, it’s xxxxx.zoeylanms.com) to CDN (xxxx.azureedge.net), and finally reaching Storage account (static website). Option 2: Using Application Gateway This option is similar to the first one, but instead of using Azure CDN to transfer requests, we will use Application Gateway to act as a web traffic load balancer and provide advanced security features. Here are the steps to set up Application Gateway: Step 1-3 in Option 1 are still needed if you don’t have your DNS provider. [Step 4] You need to have an Application Gateway resource on Azure. You could follow our official document to create an Application Gateway on Azure Portal: https://learn.microsoft.com/en-us/azure/application-gateway/quick-create-portal Note: The backend protocol used in the document example is HTTP. We can update it to HTTPS in later step. [Step 5] You can check the public IP of the Application Gateway you just created first. Then, you need to go back to the DNS zone you created earlier and add a new A record to map traffic to the Application Gateway using the public IP. [Step 6] You open the Application Gateway, go to the Backend Pools, and click the backend pool created just now. For the Target Type, you should pick "IP address or FQDN" and then put the static web site endpoint in the Target. Format: xxxxx.z7.web.core.windows.net [Step 7] In the Backend Settings, you should make sure the Backend protocol is updated to “HTTPS”. [Step 8] Then, you can create a custom health probe with a protocol in HTTPS like below to monitor the health of the resources in the backend pool. [Step 9] Before you save the new health probe, there is a test showing the connection result. Meanwhile, you can also check the backend health like the screenshot below later. Option 3: Using Azure Front Door Configuring request forwarding using Front Door is very similar to Azure CDN. You can follow our official documentation to complete the deployment. https://learn.microsoft.com/en-us/azure/storage/blobs/storage-custom-domain-name?tabs=azure-portal#using-azure-front-door Comparison Option Azure CDN Application Gateway Azure Front Door Behavior in custom domain setting Azure CDN does not have a built-in option for force HTTPS redirection. In this scenario, when accessing the website through a custom domain using HTTP, you will receive a 400 error with the error message 'AccountRequiresHttps'. However, everything works normally when accessing the webpage through a custom domain link using HTTPS. The scenarios that can be achieved with Application Gateway are more flexible because we can configure and combine the listener's protocol and backend setting's protocol separately. Therefore, whenever client accesses through either HTTP or HTTPS, by configuring the backend setting, we can forward requests to the backend storage account using HTTPS. Azure Front Door does have a built-in option for force HTTPS redirection. This can be configured in the Front Door's routing rules by selecting “Redirect all traffic to use HTTPS”. When a user tries to access the website using HTTP, Front Door will automatically redirect the request to HTTPS. Pricing Outbound data transfers are billed based on the node location from where the transfers are served, not the end user’s location. Reference: Understanding Azure CDN billing | Microsoft Learn Pricing - Content Delivery Network (CDN) | Microsoft Azure The cost of Application Gateway is mainly composed of fixed and variable charges. The fixed charge is based on a fixed hourly rate (which may vary by region), while the variable charge depends on the number of capacity units, with an hourly price for each capacity unit. Reference: Understanding pricing - Azure Application Gateway | Microsoft Learn Application Gateway Pricing | Microsoft Azure Each part of the request process is billed separately: Number of requests from client to Front Door Data transfer from Front Door edge to origin Data transfer from origin to Front Door (non-billable) Data transfer from Front Door to client Reference: Understand Azure Front Door billing | Microsoft Learn Pricing - Front Door | Microsoft Azure Performance Azure CDN offers global content delivery with low latency and high throughput. It stores cached content on the edge servers’ POP locations that are close to consuming users, thereby minimizing network latency. Reference: What is a content delivery network (CDN)? - Azure | Microsoft Learn Application Gateway provides layer 7 load balancing and web application firewall (WAF) capabilities, offers advanced traffic management features such as session affinity, redirection, and URL-based routing. It provides centralized management and monitoring of multiple backend servers. Reference: What is Azure Application Gateway | Microsoft Learn Azure Application Gateway features | Microsoft Learn Azure Front Door provides global load balancing and routing of traffic to different backend services. It unified static and dynamic delivery offered in a single tier to accelerate and scale your application through caching, SSL offload, and layer 3-4 DDoS protection. Reference: Azure Front Door | Microsoft Learn To summarize the blog, the most important thing to remember is that Azure Storage currently does not support the HTTPS Only + Custom domain mode. Therefore, we need to leverage a resource that can perform request forwarding to achieve our goal. In the previous sections, we introduced three of the most commonly used resources: Azure CDN, Application Gateway, and Azure Front Door.How to set ACLs Recursively?
Ensuring adequate access control is of utmost importance for effectively safeguarding your Azure Storage assets. The Access Control Lists (ACLs) of your Azure Storage Account assume a pivotal role in establishing permissions for both containers and blobs situated within the storage account. The inheritance of ACLs is already in place for newly generated child items beneath a parent directory. However, it's noteworthy that now you also have the capability to set, update, or remove ACLs recursively for the pre-existing child items of a parent directory, thereby eliminating the need to undertake these operations for each individual child item. In this article, I will walk you through the process of setting Azure Storage Account ACLs recursively using three different methods: PowerShell, AZ CLI and Postman. 1. PowerShell This example sets the ACL of a directory named mydirectory001 through the "Set-AzDataLakeGen2AclRecursive" command. These entries give the owning user read, write, and execute permissions, gives the owning group only read and execute permissions, and gives all others no access. The last ACL entry in this example gives a specific user (an application registration that I created) with the object ID "c0fb41ac-cd34-4380-a482-470251b089c3" read and execute permissions: $ctx = New-AzStorageContext -StorageAccountName myazurestorageaccount020 -StorageAccountKey {mykey} $acl = Set-AzDataLakeGen2ItemAclObject -AccessControlType user -Permission rwx $acl = Set-AzDataLakeGen2ItemAclObject -AccessControlType group -Permission r-x -InputObject $acl $acl = Set-AzDataLakeGen2ItemAclObject -AccessControlType other -Permission "---" -InputObject $acl $acl = Set-AzDataLakeGen2ItemAclObject -AccessControlType user -EntityId c0fb41ac-cd34-4380-a482-470251b089c3 -Permission r-x -InputObject $acl Set-AzDataLakeGen2AclRecursive -Context $ctx -FileSystem myfilesystem001 -Path mydirectory001 -Acl $acl Once the command is completely executed, you should see an output similar to the following showing the successful operations on directories, files, the failure count, and a continuation token if applicable: 2. AZ CLI This example sets the ACL of a directory named mydirectory001 through the "az storage fs access set-recursive" AZ CLI command: az storage fs access set-recursive --acl "user::rwx,group::r-x,other::---,user:c0fb41ac-cd34-4380-a482-470251b089c3:rwx" -p mydirectory001/ -f myfilesystem001 --account-name myazurestorageaccount020 --auth-mode key --account-key {mykey} 3. Postman This example sets the ACL of a directory named mydirectory001 through the proper PATCH API call in Postman: PATCH /myfilesystem001/mydirectory001?action=setAccessControlRecursive&mode=set HTTP/1.1 Authorization: Bearer (Redacted) x-ms-version: 2023-01-03 x-ms-acl: user::rwx,group::r-x,other::---,user:c0fb41ac-cd34-4380-a482-470251b089c3:r-x User-Agent: PostmanRuntime/7.32.3 Accept: */* Postman-Token: 7ef695a6-c446-4b35-a0ce-b54ba08db51a Host: myazurestorageaccount020.dfs.core.windows.net Accept-Encoding: gzip, deflate, br Connection: keep-alive Content-Length: 0 HTTP/1.1 200 OK Transfer-Encoding: chunked Server: Windows-Azure-HDFS/1.0 Microsoft-HTTPAPI/2.0 x-ms-namespace-enabled: true x-ms-request-id: fa55f148-001f-00db-76ff-ca21fc000000 x-ms-version: 2023-01-03 Date: Wed, 09 Aug 2023 20:26:05 GMT {"directoriesSuccessful":2,"failedEntries":[],"failureCount":0,"filesSuccessful":5} Recovering from failures You may come across runtime or permission-related issues while attempting to modify ACLs recursively. In the event of a permission failure, you can obtain a continuation token by configuring the --continue-on-failure parameter to false. Once you rectify the errors, you can resume the process from the point of failure by re-running the command and setting the --continuation parameter to the previously received continuation token. In the case of runtime errors (outages or connectivity issues), it's advisable to restart the process from the beginning, this is so because in such scenarios there won't be a continuation token available. It's important to note that reapplying ACLs to items won't result in any negative consequences. Even so, we suggest executing the recursive ACL process within an Azure Virtual Machine situated in the same region as your storage account, this approach will help minimizing latency and to optimize the process in general. Bonus tip! You can find this very same option through the Azure Storage Explorer tool, it's the one that says "Propagate Access Control Lists...". This feature gets the current ACLs assigned to the selected path and then executes a recursive REST API operation to assign the same permissions to the child objects: I hope that helps on identifying different options to set ACLs recursively and to save some time and effort by not having to execute a different operation per child object. References ======= Use Azure CLI to manage ACLs in Azure Data Lake Storage Gen2 https://learn.microsoft.com/en-us/azure/storage/blobs/data-lake-storage-acl-cli Use PowerShell to manage ACLs in Azure Data Lake Storage Gen2 https://learn.microsoft.com/en-us/azure/storage/blobs/data-lake-storage-acl-powershell Use Azure Storage Explorer to manage ACLs in Azure Data Lake Storage Gen2 https://learn.microsoft.com/en-us/azure/storage/blobs/data-lake-storage-explorer-aclTroubleshooting Azure Storage Deletion: Immutability, WORM, and Version Retention
One of the most common questions Azure Storage administrators ask after enabling immutable storage is: "Why can't I delete my storage account anymore?" Azure Immutable Storage is designed to provide WORM (Write Once, Read Many) protection, ensuring that data cannot be modified or deleted during a defined retention period. This capability is essential for compliance, regulatory, legal, and audit requirements, but it can be surprising when administrators later attempt to remove storage resources and discover that Azure refuses to delete them. In some environments, immutable storage policies may not have been configured directly by Azure Storage administrators. Backup or data protection solutions can automatically enable and manage retention settings, and storage administrators may not always have visibility into these externally managed policies. This article explains: Why storage accounts become undeletable Why containers sometimes appear empty but still cannot be deleted The difference between Unlocked and Locked policies How ImmutableStorageWithVersioning changes deletion behavior What happens when version retention is reduced to one day A practical step-by-step process to remove protected resources Understanding Azure Storage Immutability Azure Blob Storage supports immutability through two primary mechanisms: Time-Based Retention Policies A retention policy protects data for a fixed number of days. During this period: read are allowed, deletes are blocked and overwrites are blocked. Examples: 7 days,30 days,180 days,7 years Legal Holds Legal holds have no expiration date. The data remains protected until the hold is manually removed. Organizations commonly use legal holds when: litigation is in progress, regulatory investigations are occurring and data preservation is legally required Where Can Immutability Be Applied? Azure supports immutability at two different scopes. Scope Description Container-level Policy protects all blobs within the container Version-level Individual blob versions receive protection Version-level protection requires: Blob Versioning enabled and ImmutableStorageWithVersioning enabled Important: A storage account containing immutable data may itself become protected from deletion. What Happens When You Try to Delete Resources? Immutability blocks deletion at multiple layers. Think of it as a dependency chain: Storage Account └─ Container └─ Blob └─ Blob Version Azure will not allow deletion of a parent resource while protected child resources still exist. Scenario 1: Deleting the Storage Account You attempt to delete the storage account. Azure returns: Failed to delete storage account 'accountname'. Error: 'accountname' is protected from deletion. This occurs when at least one container still contains: a locked immutability policy, a legal hold or Immutable blob versions Before the account can be deleted: every user container must be removed, every blob must be removed, every blob version must be removed and every snapshot must be removed What About System Containers? System containers do not block deletion.These are managed automatically by Azure. The containers that matter are user-created containers. Examples: $logs, $blobchangefeed and $web Scenario 2: Deleting a Container You select Delete Container and Azure returns: The storage account container is protected from deletion due to ImmutableStorageWithVersioning. Even when, the container appears empty and the latest blobs have been deleted. Deletion may still fail. Why? Previous blob versions still exist. This is one of the most common causes of confusion. The Azure Portal normally displays: Only Show Active Blobs As a result, previous versions remain hidden. The container looks empty, but Azure still sees historical versions, snapshots and soft-deleted objects Although the container may appear empty at first glance, it is not actually empty. When "Show active and Deleted blobs" is enabled, the container displays all blobs that are currently in either the active or deleted state. Therefore, deletion remains blocked. Deleting containers with immutability (WORM) policies Container deletion may fail if an Immutable blob storage policy is applied at the container or version level. These policies keep blobs in a non-modifiable, delete-protected state (WORM) for the duration of the defined retention interval. The behavior depends on the policy State: Unlocked: the policy can still be edited or removed. You must delete the immutability policy reference from the container before deleting it. Once removed, the container and its blobs can be deleted normally. Locked: the policy cannot be deleted or shortened. You must wait until the retention interval expires for all affected blobs. Until that time elapses, the blobs, the container, and the storage account cannot be deleted. Scenario 3: Deleting Individual Blobs The retention policy prevents deletion of the blob regardless of whether the Delete option appears available in the user interface. Any delete request is rejected by the Storage service until the retention period expires. This indicates that an active retention policy is protecting the blob. Until retention expires: Delete, overwrite and modification are blocked. Unlocked Policies Think of this state as a testing phase. Protection is active, but administrators retain management flexibility. You can, Increase or decrease the retention period, or delete the policy. Once the policy is removed, the data becomes deletable. This allows organizations to validate configurations before committing to compliance requirements. Locked Policies A locked policy represents a compliance commitment. After locking: the policy cannot be deleted, retention cannot be reduced or extended. Most importantly: No user, administrator, or Microsoft Support engineer can bypass or remove a locked retention policy before expiration. i.e If retention is configured for 180 days, deletion becomes possible only after those 180 days have passed. The Hidden Detail: ImmutableStorageWithVersioning A common misunderstanding is believing that removing an unlocked retention policy completely disables immutability. It does not. Once ImmutableStorageWithVersioning is enabled: The feature itself cannot be disabled. Even if every unlocked policy is deleted, Azure still requires all versions to be removed before containers and storage accounts can be deleted. Why Version Retention Matters Blob versioning introduces another important behavior. When you modify version retention and reduce it to a lower value, for example: Once the version retention period is reduced (for example, from 180 days to 1 day), Azure automatically updates or creates the corresponding Lifecycle Management rule responsible for cleaning up old blob versions. At this stage, administrators can manually delete any remaining base blobs, snapshots, or versions that are already eligible for deletion. For versions that still exist and exceed the newly configured retention threshold: No additional manual action is required, the Lifecycle Management rule will automatically identify them as eligible for deletion and all versions older than the configured retention period are processed during subsequent Lifecycle Management executions. Example, If version retention is changed from 180 days to 1 day: Versions created more than 1 day ago immediately become eligible for cleanup, the Lifecycle Management engine evaluates those versions during its next execution cycle and once the versions are removed, container deletion can proceed provided no other immutable objects remain. Important Consideration Lifecycle Management is an asynchronous service. Therefore, after reducing retention: The configuration change is applied immediately, older versions become eligible for deletion immediately and actual version removal may take 24-48 hours or longer, depending on the Lifecycle Management execution schedule. For this reason, it is normal to see containers remain undeletable for a period of time after retention settings are modified. Administrators should allow sufficient time for Lifecycle Management to complete cleanup of all versions that exceed the configured retention threshold before attempting to delete the container or storage account. Azure lifecycle rule created automatically after changing the versioning retention. Many administrators are unaware that this happens behind the scenes. Code View for automatic rule created The purpose is simple: Remove older blob versions once they exceed the configured age. Reducing the retention period does not instantly delete existing versions. Instead, it makes existing versions eligible for deletion by Lifecycle Management, which removes them asynchronously during future processing cycles. Common Misconception Many customers assume: "Only versions created after I changed the setting will be deleted." This is incorrect. Azure evaluates every version independently using its creation timestamp. If a version is already older than one day when the rule runs: It becomes eligible for deletion. There is no grandfathering mechanism. Another Important Detail: Deletion Is Not Immediate Lifecycle Management follows an asynchronous process. Typical behavior: Action Expected Time New rule created Immediate Rule evaluated Within 24–48 hours Version cleanup begins During next lifecycle execution Because of this delay, containers may remain undeletable for some time even after retention settings are updated. Recommended Cleanup Process When attempting to remove immutable storage resources: Step 1 — Inventory all protection mechanisms: Account-level immutability, Container-level policies, Blob-level policies, Legal holds. Step 2 — Determine policy state: Locked or Unlocked Step 3 — Remove unlocked policies. Step 4 — Clear legal holds. Step 5 — Review versioning and soft delete settings. Step 6 — Remove all blobs, snapshots, and versions. Step 7 — Allow Lifecycle Management to remove old versions. Step 8 — Delete containers. Step 9 — Delete the storage account. Important Note: The deletion order is always blobs → container → storage account. A parent resource cannot be removed while protected children remain, so if any blob is still under a Locked policy (or an active legal hold), the entire process is blocked until retention expires or the hold is cleared. Key Takeaways ✅ Immutable storage protects data against deletion and modification. ✅ Locked policies cannot be bypassed, shortened, or removed. ✅ Legal holds must be manually cleared. ✅ Containers that appear empty may still contain hidden versions. ✅ ImmutableStorageWithVersioning cannot be turned off once enabled. ✅ Blob version retention automatically creates a Lifecycle Management rule. ✅ Storage account deletion requires every protected object underneath it to be removed first. References https://learn.microsoft.com/azure/storage/blobs/immutable-storage-overview https://learn.microsoft.com/azure/storage/blobs/immutable-policy-configure-container-scope https://learn.microsoft.com/azure/storage/blobs/immutable-policy-configure-version-scope https://learn.microsoft.com/azure/storage/blobs/data-protection-overview https://learn.microsoft.com/azure/storage/blobs/lifecycle-management-overviewLifecycle Management of Blobs (Deletion) using Automation Tasks
Background: We often encounter scenarios where we need to delete blobs that have been idle in a storage account for an extended period. For a small number of blobs, deletion can be handled easily using the Azure Portal, Storage Explorer, or inline scripts such as PowerShell or Azure CLI. However, in most cases, we deal with a large volume of blobs, making manual deletion impractical. In such situations, it's essential to leverage automation tools to streamline the deletion process. One effective option is using Automation Tasks, which can help schedule and manage blob deletions efficiently. Note: Behind the scenes, an automation task is actually a logic app resource that runs a workflow. So, the Consumption pricing model of logic-app applies to automation tasks. Scenario’s where “Automation Tasks” are helpful: You have a requirement to automate deletion of blobs which are older than a specific time, in days, weeks or months. You don’t want to put in much manual effort rather have simple UI based steps to achieve your goal You have System containers, and you want to action on it. We have “LCM (Life Cycle management)” which too can be leveraged by users to automation deletion of older blobs; however LCM cannot be used to delete blobs from System containers. You have to work on page blobs. Setup “Automation Tasks”: Let’s walk through on how to achieve our goal. Navigate to the desired storage account and scroll down to the “Automation” section and select the “Tasks” blade and then click on “Add Task” from the top panel or bottom panel (highlighted in image). On the next page click the “Select” (highlighted image) The new page which opens up should look as below, however there isn’t anything we are doing. So let’s just click on the “Next : Configure” (highlighted in image) and move to the next screen. The new page opens needs to be filled as per your requirement. I have added a sample. You can also use it on your own containers as well. 'sample' is a folder inside container '$web' The “Expiration Age” field means that Blobs older than these number of days needs be deleted. In above screenshot, blobs older than 180 days would be deleted. Similarly, we can configure values in weeks or months as well. Once we are through with the steps proceed with creation of the task. Once task is created it looks as below: You can click on the “View Run” to see run history. In-case you want to modify the task, click on your tasks name. For example in above screenshot I can modify by clicking “mytask” link and re-configure the task. Now this isn’t sufficient. We will update some of the steps which were used to create the Logic-app. Hence we would need to edit some steps and save those before re-running the app. a) Go the logic app and navigate to the “Logic App Designer” blade b) Now click on the “+” sign as shown below and “Add an Action” c) Once the new page opens up, search for “List Blobs (v2)” and select it d) Choose the “Enter custom value” and enter your storage account name e) The values would like as shown below f) Now let's navigate to the “For Each” condition g) We need to delete the “Delete blob” too and replace with “Delete blob (V2)” h) The “Delete Blob (V2)” looks like as below i) With all steps ready, lets save the logic app and click on “Run”. You should observe the run passing successfully. Impact due to Firewall: The above steps for works when your storage account is configured for public access. However, when firewall is enabled, you would need to provide the necessary permissions, else you are going to encounter 403 "Authorization Failure" errors. There would be no issue to create the task, but you will see failures when you check the runs. Example: To overcome this limitation, you need to navigate to your logic app and generate a managed identity for the app and provide the identity “Storage Blob Data Contributor” role. Step1. Enable Managed Identity: In Azure Portal, go to your Logic App resource. Under Settings, select Identity. In the Identity pane, under System assigned, select On and Save. This step registers the system-assigned identity with Microsoft Entra ID, represented by an object ID. Step2. Assign Necessary Role: Open the Azure Storage Account in Azure Portal. Select Access control (IAM) > Add > Add role assignment. Assign a role like 'Storage Blob Data Contributor', which includes write access for blobs in an Azure Storage container, to the managed identity. Under Assign access to, select Managed identity > Add members, and choose your Logic App's identity Save and refresh and you see the new role configured to your storage account Remember that, if the Storage Account and logic app are in different region you should add another step in the firewall of storage account. You need to whitelist the logic app instance in “Resource instances” list as shown below: Conclusion: The multiple ways to action on blobs are provided for your convenience. Depending on your requirement, feasibility & other factors like comfortability with the feature or pricing too would certainly influence your decisions. However, in-case you want to action upon System containers like $logs or $web, “Automation Tasks” are one of the most helpful feature which you can use and achieve your goal. Note: At the time of writing this blog this feature is still in preview. So ensure to check if there are any limitations which might impact you before implementing it in your Production environment. References: Create automation tasks to manage and monitor Azure resources - Azure Logic Apps | Microsoft Learn Optimize costs by automatically managing the data lifecycle - Azure Blob Storage | Microsoft Learn
