This blog introduces how to automatically delete aged page blobs using Function App. This method also supports adding more filter conditions, for example, filtering blobs using wildcard which is not supported in the Lifecycle Management rule (supports prefix filter only).

Background:

In Azure Storage, we have Lifecycle Management which offers a rule-based policy that you can use to expire data at the end of the data lifecycle. However, the use cases of Lifecycle Management are limited. Currently, Lifecycle Management policies are only supported for block blobs and append blobs in general-purpose v2, premium block blob, and Blob Storage accounts.

For page blobs, the Azure Storage Lifecycle Management does not support at this moment, but you can achieve the same goal by using the Function App.

This blog introduces how to automatically delete aged page blobs using Function App. This method also supports adding more filter conditions, for example, filtering blobs using wildcard which is not supported in the Lifecycle Management rule (supports prefix filter only).

Reference: https://docs.microsoft.com/en-us/azure/storage/blobs/lifecycle-management-overview 

Steps to Configure:

Step 1: Create a Function App

To create a Function App, sign in to the Azure Portal and search for Function App.

Click on Create and configure the settings for your new Function App.

For Runtime stack, select PowerShell Core.

Click on the Review + create button on the bottom of the page and then click on Create again on the next page.

Step 2: Install Required Module – Az.Storage

Select the Function App that you just create.

\n
1. Click on the App files.
\n
2. Select requirements.psd1 in the dropdown list.
\n
3. Add a new line in the code: 'Az.Storage' = '3.11.0'.
\n
4. Save the change.
\n

Step 3: Create a Timer Trigger Function in the Function App

Click on Functions and create a new Timer trigger function.

Step 4: Set the proper configuration for the Timer Trigger Function

Click on Code + Test. This opens the code editor for you.

Select the run.ps1 in the dropdown list.

Copy and paste the sample code below into the editor and modify the parameters’ value accordingly.

Sample code 1: Delete page blobs that haven’t been modified in the past 24 hours.

param($Timer)

$lifecycle = [DateTime]::UtcNow.AddHours(-24)     # 1 day

$context = New-AzStorageContext -StorageAccountName \"YourStorageAccountName\" -StorageAccountKey \"YourStorageAccountKey—CanBeFoundInAccessKeyofStorageAccount\"

Get-AzStorageBlob -Container \"YourContainerName\" -Context $context | Where-Object { $_.LastModified.UtcDateTime -lt $lifecycle -and $_.BlobType -eq \"PageBlob\" -and $_.Name -like \"*.vhd\"} | Remove-AzStorageBlob

Reference: https://docs.microsoft.com/en-us/answers/questions/46821/how-can-we-automatically-delete-page-blobs-from-az.html

You can apply the automatic deletion to specific page blobs by limiting the filters. In the above sample code, it delete page blobs that end with “.vhd” and were not modified in the past 24 hours.

For Account Key, you can find it in your storage account, under the Access keys, as the screenshot shown below.

Another advantage of using the Timer trigger code is that you can use the wildcard to filter items by different criteria, such as prefix, suffix, substrings.

The following sample code is used to change the access tier for block blobs to “Archive” whose last modified date is earlier than 2 days ago and the file name having “blob” in between.

Sample code 2: Change the access tier of block blobs using wildcards that haven’t been modified in the past 2 days.

param($Timer)

$lifecycle = [DateTime]::UtcNow.AddHours(-48) #2 days

#Select the blob from a container
$blob =  Get-AzStorageBlob -Container \" YourContainerName\" -Context $context | Where-Object { $_.LastModified.UtcDateTime -gt $lifecycle -and $_.BlobType -eq \"BlockBlob\" -and $_.Name -like \"*blob*\"}

#Change the blob's access tier to archive
$blob.ICloudBlob.SetStandardBlobTier(\"Archive\")

Select the function.json file and modify the schedule value based on your requirement.

You can set the frequency whatever you want the function to be triggered with, for example, once a day.

Here are some examples of NCRONTAB expressions.

Reference: https://docs.microsoft.com/en-us/azure/azure-functions/functions-bindings-timer?tabs=csharp#ncrontab-expressions

Step 5: Monitor the Timer Trigger or Disable it

Click on Monitor on the left menu. This opens the Invocation Traces by which you can monitor how the Timer trigger function works.

Click on Disable to stop the Timer Trigger function if needed.

Conclusion:

By using the Timer Trigger function explained above, you can achieve two goals, which can be used as a good supplement to the Lifecycle Management.

\n
1. You can automatically manage the page blob data lifecycle.
\n
2. You can add filters using wildcards (not only prefix filters) to manage your data.
\n

Background:

In Azure Storage, we have Lifecycle Management which offers a rule-based policy that you can use to expire data at the end of the data lifecycle. However, the use cases of Lifecycle Management are limited. Currently, Lifecycle Management policies are only supported for block blobs and append blobs in general-purpose v2, premium block blob, and Blob Storage accounts.

For page blobs, the Azure Storage Lifecycle Management does not support at this moment, but you can achieve the same goal by using the Function App.

This blog introduces how to automatically delete aged page blobs using Function App. This method also supports adding more filter conditions, for example, filtering blobs using wildcard which is not supported in the Lifecycle Management rule (supports prefix filter only).

Reference: https://docs.microsoft.com/en-us/azure/storage/blobs/lifecycle-management-overview 

Steps to Configure:

Step 1: Create a Function App

To create a Function App, sign in to the Azure Portal and search for Function App.

Click on Create and configure the settings for your new Function App.

For Runtime stack, select PowerShell Core.

Click on the Review + create button on the bottom of the page and then click on Create again on the next page.

Step 2: Install Required Module – Az.Storage

Select the Function App that you just create.

\n
1. Click on the App files.
\n
2. Select requirements.psd1 in the dropdown list.
\n
3. Add a new line in the code: 'Az.Storage' = '3.11.0'.
\n
4. Save the change.
\n

Step 3: Create a Timer Trigger Function in the Function App

Click on Functions and create a new Timer trigger function.

Step 4: Set the proper configuration for the Timer Trigger Function

Click on Code + Test. This opens the code editor for you.

Select the run.ps1 in the dropdown list.

Copy and paste the sample code below into the editor and modify the parameters’ value accordingly.

Sample code 1: Delete page blobs that haven’t been modified in the past 24 hours.

param($Timer)

$lifecycle = [DateTime]::UtcNow.AddHours(-24)     # 1 day

$context = New-AzStorageContext -StorageAccountName \"YourStorageAccountName\" -StorageAccountKey \"YourStorageAccountKey—CanBeFoundInAccessKeyofStorageAccount\"

Get-AzStorageBlob -Container \"YourContainerName\" -Context $context | Where-Object { $_.LastModified.UtcDateTime -lt $lifecycle -and $_.BlobType -eq \"PageBlob\" -and $_.Name -like \"*.vhd\"} | Remove-AzStorageBlob

Reference: https://docs.microsoft.com/en-us/answers/questions/46821/how-can-we-automatically-delete-page-blobs-from-az.html

You can apply the automatic deletion to specific page blobs by limiting the filters. In the above sample code, it delete page blobs that end with “.vhd” and were not modified in the past 24 hours.

For Account Key, you can find it in your storage account, under the Access keys, as the screenshot shown below.

Another advantage of using the Timer trigger code is that you can use the wildcard to filter items by different criteria, such as prefix, suffix, substrings.

The following sample code is used to change the access tier for block blobs to “Archive” whose last modified date is earlier than 2 days ago and the file name having “blob” in between.

Sample code 2: Change the access tier of block blobs using wildcards that haven’t been modified in the past 2 days.

param($Timer)

$lifecycle = [DateTime]::UtcNow.AddHours(-48) #2 days

#Select the blob from a container
$blob =  Get-AzStorageBlob -Container \" YourContainerName\" -Context $context | Where-Object { $_.LastModified.UtcDateTime -gt $lifecycle -and $_.BlobType -eq \"BlockBlob\" -and $_.Name -like \"*blob*\"}

#Change the blob's access tier to archive
$blob.ICloudBlob.SetStandardBlobTier(\"Archive\")

Select the function.json file and modify the schedule value based on your requirement.

You can set the frequency whatever you want the function to be triggered with, for example, once a day.

Here are some examples of NCRONTAB expressions.

Reference: https://docs.microsoft.com/en-us/azure/azure-functions/functions-bindings-timer?tabs=csharp#ncrontab-expressions

Step 5: Monitor the Timer Trigger or Disable it

Click on Monitor on the left menu. This opens the Invocation Traces by which you can monitor how the Timer trigger function works.

Click on Disable to stop the Timer Trigger function if needed.

Conclusion:

By using the Timer Trigger function explained above, you can achieve two goals, which can be used as a good supplement to the Lifecycle Management.

\n
1. You can automatically manage the page blob data lifecycle.
\n
2. You can add filters using wildcards (not only prefix filters) to manage your data.
\n

This blog introduces how to automatically delete aged page blobs using Function App. This method also supports adding more filter conditions, for example, filtering blobs using wildcard which is not supported in the Lifecycle Management rule (supports prefix filter only).

Hi,

Can I use the same below sample code to change the page blobs to Archive access tier. 

Sample code 2: Change the access tier of block blobs using wildcards that haven’t been modified in the past 2 days.

param($Timer)

$lifecycle = [DateTime]::UtcNow.AddHours(-48) #2 days

#Select the blob from a container
$blob =  Get-AzStorageBlob -Container \" YourContainerName\" -Context $context | Where-Object { $_.LastModified.UtcDateTime -gt $lifecycle -and $_.BlobType -eq \"BlockBlob\" -and $_.Name -like \"*blob*\"}

#Change the blob's access tier to archive
$blob.ICloudBlob.SetStandardBlobTier(\"Archive\")

Could that be done with Azure Automation account? Any pros /cons with function?

It's useful, thanks for you work zoey!

I wanted to confirm if we have any option to manage page blob by life cycle management policy. Our requirement to use the condition \"if the page blob (snapshots) are not modified in last 7 days then it will automatically delete as per the Life cycle management policy\". The above post I can see on October 21, So is there any further update or any other Microsoft document where it is clearly mentioned this feature is not supported for page blob. As of now we can see only block blob and append blob when we set the rule. Also let us know, If the feature can be enabled very soon.

Regards,

Sourav Mitra