Lifecycle Management for Page Blob and Block Blob using Function App.

Background:

 

In Azure Storage, we have Lifecycle Management which offers a rule-based policy that you can use to expire data at the end of the data lifecycle. However, the use cases of Lifecycle Management are limited. Currently, Lifecycle Management policies are only supported for block blobs and append blobs in general-purpose v2, premium block blob, and Blob Storage accounts.

 

For page blobs, the Azure Storage Lifecycle Management does not support at this moment, but you can achieve the same goal by using the Function App.

 

This blog introduces how to automatically delete aged page blobs using Function App. This method also supports adding more filter conditions, for example, filtering blobs using wildcard which is not supported in the Lifecycle Management rule (supports prefix filter only).

 

Reference: https://docs.microsoft.com/en-us/azure/storage/blobs/lifecycle-management-overview 

 

 

Steps to Configure:

 

Step 1: Create a Function App

 

To create a Function App, sign in to the Azure Portal and search for Function App.

 

Click on Create and configure the settings for your new Function App.

 

For Runtime stack, select PowerShell Core.

 

Click on the Review + create button on the bottom of the page and then click on Create again on the next page.

 

 

Step 2: Install Required Module – Az.Storage

 

Select the Function App that you just create.

1. Click on the App files.
2. Select requirements.psd1 in the dropdown list.
3. Add a new line in the code: 'Az.Storage' = '3.11.0'.
4. Save the change.

 

 

Step 3: Create a Timer Trigger Function in the Function App

 

Click on Functions and create a new Timer trigger function.

 

 

 

Step 4: Set the proper configuration for the Timer Trigger Function

 

Click on Code + Test. This opens the code editor for you.

 

Select the run.ps1 in the dropdown list.

Copy and paste the sample code below into the editor and modify the parameters’ value accordingly.

 

Sample code 1: Delete page blobs that haven’t been modified in the past 24 hours.

param($Timer)

$lifecycle = [DateTime]::UtcNow.AddHours(-24)     # 1 day

$context = New-AzStorageContext -StorageAccountName "YourStorageAccountName" -StorageAccountKey "YourStorageAccountKey—CanBeFoundInAccessKeyofStorageAccount"

Get-AzStorageBlob -Container "YourContainerName" -Context $context | Where-Object { $_.LastModified.UtcDateTime -lt $lifecycle -and $_.BlobType -eq "PageBlob" -and $_.Name -like "*.vhd"} | Remove-AzStorageBlob

Reference: https://docs.microsoft.com/en-us/answers/questions/46821/how-can-we-automatically-delete-page-blobs-from-az.html

 

You can apply the automatic deletion to specific page blobs by limiting the filters. In the above sample code, it delete page blobs that end with “.vhd” and were not modified in the past 24 hours.

 

For Account Key, you can find it in your storage account, under the Access keys, as the screenshot shown below.

 

 

Another advantage of using the Timer trigger code is that you can use the wildcard to filter items by different criteria, such as prefix, suffix, substrings.

The following sample code is used to change the access tier for block blobs to “Archive” whose last modified date is earlier than 2 days ago and the file name having “blob” in between.

 

Sample code 2: Change the access tier of block blobs using wildcards that haven’t been modified in the past 2 days.

param($Timer)

$lifecycle = [DateTime]::UtcNow.AddHours(-48) #2 days

#Select the blob from a container
$blob =  Get-AzStorageBlob -Container " YourContainerName" -Context $context | Where-Object { $_.LastModified.UtcDateTime -gt $lifecycle -and $_.BlobType -eq "BlockBlob" -and $_.Name -like "*blob*"}

#Change the blob's access tier to archive
$blob.ICloudBlob.SetStandardBlobTier("Archive")

 

Select the function.json file and modify the schedule value based on your requirement.

 

You can set the frequency whatever you want the function to be triggered with, for example, once a day.

Here are some examples of NCRONTAB expressions.

0 */5 * * * *	once every five minutes
0 0 * * * *	once at the top of every hour
0 0 */2 * * *	once every two hours
0 30 9 * * *	at 9:30 AM every day

Reference: https://docs.microsoft.com/en-us/azure/azure-functions/functions-bindings-timer?tabs=csharp#ncrontab-expressions

 

 

Step 5: Monitor the Timer Trigger or Disable it

 

Click on Monitor on the left menu. This opens the Invocation Traces by which you can monitor how the Timer trigger function works.

 

Click on Disable to stop the Timer Trigger function if needed.

 

 

Conclusion:

By using the Timer Trigger function explained above, you can achieve two goals, which can be used as a good supplement to the Lifecycle Management.

1. You can automatically manage the page blob data lifecycle.
2. You can add filters using wildcards (not only prefix filters) to manage your data.