Automate the backup of your block blobs with Azure Tags and Azure Automation Account.

Introduction

Hi folks! My name is Felipe Binotto, Cloud Solution Architect, based in Australia.

This post is about how you can automate the backup of Block Blob Storage using Azure Backup Vault (not to be confused with Azure Recovery Service Vault). I specifically mention Block Blob because append and page blobs are not supported.

By automating the backup process of your Block Blob Storage, you can rest assured that your data is safe and secure in case of unexpected data loss.

Before we start, let us look at the solution’s high-level design.

High-Level Design

The following is a similar HLD (high-level design) I have done for a customer recently.

A few things to note from the above diagram:

\n
• The backup for Block Blobs is stored in the Storage Account, meaning there is no traffic flowing between the Storage Account and the Backup Vault. Therefore, all our Backup Vaults are on the Hub. We only need 1 Backup Vault per region.
\n
• The identity of the Backup Vault requires the Storage Account Backup Contributor role on the Storage Accounts that will be backed up.
\n
• We will use an Automation Account to automate the backup based on a tag on the Storage Account.
\n
• There is no schedule for the Backup policies because the backup is continuous. Only retention must be specified.
\n
• We will use REST API calls to create 2 different retention policies as you cannot set them from the portal.
\n

Pre-Requisites

Before we begin, please ensure that you have the following prerequisites in place:

\n
• Azure Subscription
\n
• Automation Account
\n
• Storage Account
\n
• Backup Vault
\n

High-Level Steps

The process of automating the backup of Block Blobs in the storage account can be broken down in a few steps, which we will go through in the following order:

\n
1. Create a User Assigned identity for your Automation Account
\n
2. Assigning the role of Backup Contributor for the identity
\n
3. Create Backup policies.
\n
4. Create a Runbook.
\n

Getting Started

To start off, let us create a User Assigned Identity which we will assign to our Automation Account and grant the Backup Contributor role so that we can create the Backup policies.

$id = (New-AzUserAssignedIdentity -Name backupautomation -ResourceGroupName REPLACE_WITH_YOUR_RG –Location australiaeast).principalId

Now let us assign the Backup Contributor role to the identity.

New-AzRoleAssignment -ObjectId $id `\n-RoleDefinitionName 'Backup Contributor' `\n-Scope /subscriptions/<subscriptionId>

And assign the identity to the Automation Account.

# Get the automation account\n$automationAccount = Get-AzAutomationAccount -ResourceGroupName REPLACE_WITH_YOUR_RG -Name REPLACE_WITH_YOUR_AA\n\n# Assign the identity to the automation account\n$automationAccount | Set-AzAutomationAccount -Identity $id

Next, we will create two Backup policies. One for our production environment which will have a retention of 30 days and another one for our non-production environment which will have a retention of 10 days.

First, we must create a payload with the policy details. We will start with the production policy.

$payload = @\"\n{\n            \"properties\": {\n                \"datasourceTypes\": [\n                \"Microsoft.Storage/storageAccounts/blobServices\"\n                ],\n                \"objectType\": \"BackupPolicy\",\n                \"policyRules\": [\n                    {\n                        \"name\": \"Default\",\n                        \"objectType\": \"AzureRetentionRule\",\n                        \"isDefault\": true,\n                        \"lifecycles\": [\n                        {\n                        \"deleteAfter\": {\n                        \"duration\": \"P30D\",\n                        \"objectType\": \"AbsoluteDeleteOption\"\n                        },\n                        \"sourceDataStore\": {\n                        \"dataStoreType\": \"OperationalStore\",\n                        \"objectType\": \"DataStoreInfoBase\"\n                        }\n                    }\n                ]\n            }\n        ]\n    }\n}\n\"@@

Now we can invoke the REST method to create the policy.

Invoke-AzRestMethod -SubscriptionId $MySubID -Method Put `\n-ResourceProviderName \"Microsoft.DataProtection/backupVaults/$vaultName /backupPolicies/$policyName\" `\n-ApiVersion 2021-01-01 -Payload $payload -ResourceGroupName $MyRG

Replace $MySubID with the subscription ID where the Backup Vault is located, replace $vaultName with your Backup Vault name, replace $policyName with the policy name you want (e.g., Production) and replace $MyRG with the Backup Vault resource group name.

Now let repeat the steps for non-production but change the duration in the $payload with value of “P10D”.

$payload = @\"\n{\n            \"properties\": {\n                \"datasourceTypes\": [\n                \"Microsoft.Storage/storageAccounts/blobServices\"\n                ],\n                \"objectType\": \"BackupPolicy\",\n                \"policyRules\": [\n                    {\n                        \"name\": \"Default\",\n                        \"objectType\": \"AzureRetentionRule\",\n                        \"isDefault\": true,\n                        \"lifecycles\": [\n                        {\n                        \"deleteAfter\": {\n                        \"duration\": \"P10D\",\n                        \"objectType\": \"AbsoluteDeleteOption\"\n                        },\n                        \"sourceDataStore\": {\n                        \"dataStoreType\": \"OperationalStore\",\n                        \"objectType\": \"DataStoreInfoBase\"\n                        }\n                    }\n                ]\n            }\n        ]\n    }\n}\n\"@@

Re-run the REST method but provide a different name for the $policyName (e.g., Non-Production)

Invoke-AzRestMethod -SubscriptionId $MySubID -Method Put `\n-ResourceProviderName \"Microsoft.DataProtection/backupVaults//$vaultName /backupPolicies/$policyName\" `\n-ApiVersion 2021-01-01 -Payload $payload -ResourceGroupName $MyRG

OK, now we are ready to create the Runbook which will automate the backup process.

#Ensures you do not inherit an AzContext in your runbook\n\nDisable-AzContextAutosave -Scope Process\n\n#Connect to Azure with user-assigned managed identity\nConnect-AzAccount -Identity -AccountId #ReplaceWithYourIdentityID\n\n#Set variables\n$vaultName = #ReplaceWithYourVaultName\n$resourceGroupName = #ReplaceWithYourRG\n\n#Gets the list of all subscriptions\n$subscriptions = Get-AzSubscription\n\n$query = \"resources | where type in~ ('microsoft.storage/StorageAccounts') | where tags.Environment != '' and location == 'australiaeast'\"\n\n$storages = Search-AzGraph -Subscription $subscriptions -Query $query\n\n$query = \"RecoveryServicesResources | where type in~ ('microsoft.DataProtection/backupVaults/backupInstances')\"\n\n$backups = Search-AzGraph -Subscription $subscriptions -Query $query\n\n$npePolicy = Get-AzDataProtectionBackupPolicy -ResourceGroupName $resourceGroupName -VaultName $vaultName -Name \"Non-Production\"\n\n$prdPolicy = Get-AzDataProtectionBackupPolicy -ResourceGroupName $resourceGroupName -VaultName $vaultName -Name \"Production\"\n\nforeach ($storage in $storages) {\n\n    $protected = $backups | Where-Object Name -match $storage.Name\n\n    if (!$protected -or $protected.properties.currentProtectionState -ne \"ProtectionConfigured\") {\n\n        $vault = Get-AzDataProtectionBackupVault -ResourceGroupName $resourceGroupName -VaultName $vaultName\n\n        if ($storage.tags.Environment -eq \"Non-Production\") {\n\n            $instance = Initialize-AzDataProtectionBackupInstance -DatasourceType AzureBlob -DatasourceLocation $vault.Location -PolicyId $npePolicy.id -DatasourceId $storage.ResourceId\n\n            New-AzDataProtectionBackupInstance -ResourceGroupName $resourceGroupName -VaultName  $vaultName -BackupInstance $instance -NoWait\n        }\n\n        elseif ($storage.tags.Environment -eq \"Production\") {\n\n            $instance = Initialize-AzDataProtectionBackupInstance -DatasourceType AzureBlob -DatasourceLocation $vault.Location -PolicyId $prdPolicy.id -DatasourceId $storage.ResourceId\n\n            New-AzDataProtectionBackupInstance -ResourceGroupName $resourceGroupName -VaultName  $vaultName -BackupInstance $instance -NoWait\n\n        }\n        else {\n            Write-Output \"Invalid value provided for tag Environment. Values are: Non-Production or Production\"\n        }      \n    }\n}

A few values you must replace in the script above;

\n
• Replace #ReplaceWithYourIdentityID with the identity we created in an earlier step
\n
• Replace #ReplaceWithYourVaultName with the Backup Vault Name
\n
• Replace #ReplaceWithYourRG with the Resource Group of the Backup Vault
\n

In addition, a few notes about the script;

\n
• We are only querying for Storage Accounts in the Australia East region. You could have a single automation account querying all regions, but I prefer this way to distribute the load
\n
• We are checking if the backup already exists for the retrieved Storage Accounts and if does, we do not do anything
\n
• We have defined the Tag name on the Storage Accounts to be Environment and the value to be either Production or Non-Production
\n

Finally, as a last step, you can create the required tag in one of your Storage Accounts and run the runbook.

Once you are satisfied with the results, you can create a schedule for your Runbook and tag other Storage Accounts to automatically enable backup for Block Blobs.

Conclusion

In conclusion, automating the backup process of your Block Blob Storage is a crucial step in ensuring the safety and security of your data in case of unexpected data loss. By following the steps outlined in this post, you can set up an Azure Backup Vault to continuously back up your Block Blob Storage, without any traffic flowing between the Storage Account and the Backup Vault.

Additionally, creating an Automation Account, assigning the Backup Contributor role, and creating Backup policies can help simplify the backup process. Once you have completed these steps, you can easily run the Runbook to enable automatic backup for Block Blobs in your Storage Accounts.

Overall, this solution is a reliable and efficient way to ensure the safety of your data in the cloud.

I hope this was informative to you and thanks for reading!

Disclaimer

The sample scripts are not supported under any Microsoft standard support program or service. The sample scripts are provided AS IS without warranty of any kind. Microsoft further disclaims all implied warranties including, without limitation, any implied warranties of merchantability or of fitness for a particular purpose. The entire risk arising out of the use or performance of the sample scripts and documentation remains with you. In no event shall Microsoft, its authors, or anyone else involved in the creation, production, or delivery of the scripts be liable for any damages whatsoever (including, without limitation, damages for loss of business profits, business interruption, loss of business information, or other pecuniary loss) arising out of the use of or inability to use the sample scripts or documentation, even if Microsoft has been advised of the possibility of such damages.

Introduction

Hi folks! My name is Felipe Binotto, Cloud Solution Architect, based in Australia.

This post is about how you can automate the backup of Block Blob Storage using Azure Backup Vault (not to be confused with Azure Recovery Service Vault). I specifically mention Block Blob because append and page blobs are not supported.

By automating the backup process of your Block Blob Storage, you can rest assured that your data is safe and secure in case of unexpected data loss.

Before we start, let us look at the solution’s high-level design.

High-Level Design

The following is a similar HLD (high-level design) I have done for a customer recently.

A few things to note from the above diagram:

\n
• The backup for Block Blobs is stored in the Storage Account, meaning there is no traffic flowing between the Storage Account and the Backup Vault. Therefore, all our Backup Vaults are on the Hub. We only need 1 Backup Vault per region.
\n
• The identity of the Backup Vault requires the Storage Account Backup Contributor role on the Storage Accounts that will be backed up.
\n
• We will use an Automation Account to automate the backup based on a tag on the Storage Account.
\n
• There is no schedule for the Backup policies because the backup is continuous. Only retention must be specified.
\n
• We will use REST API calls to create 2 different retention policies as you cannot set them from the portal.
\n

Pre-Requisites

Before we begin, please ensure that you have the following prerequisites in place:

\n
• Azure Subscription
\n
• Automation Account
\n
• Storage Account
\n
• Backup Vault
\n

High-Level Steps

The process of automating the backup of Block Blobs in the storage account can be broken down in a few steps, which we will go through in the following order:

\n
1. Create a User Assigned identity for your Automation Account
\n
2. Assigning the role of Backup Contributor for the identity
\n
3. Create Backup policies.
\n
4. Create a Runbook.
\n

Getting Started

To start off, let us create a User Assigned Identity which we will assign to our Automation Account and grant the Backup Contributor role so that we can create the Backup policies.

Now let us assign the Backup Contributor role to the identity.

And assign the identity to the Automation Account.

Next, we will create two Backup policies. One for our production environment which will have a retention of 30 days and another one for our non-production environment which will have a retention of 10 days.

First, we must create a payload with the policy details. We will start with the production policy.

Now we can invoke the REST method to create the policy.

Replace $MySubID with the subscription ID where the Backup Vault is located, replace $vaultName with your Backup Vault name, replace $policyName with the policy name you want (e.g., Production) and replace $MyRG with the Backup Vault resource group name.

Now let repeat the steps for non-production but change the duration in the $payload with value of “P10D”.

Re-run the REST method but provide a different name for the $policyName (e.g., Non-Production)

OK, now we are ready to create the Runbook which will automate the backup process.

A few values you must replace in the script above;

\n
• Replace #ReplaceWithYourIdentityID with the identity we created in an earlier step
\n
• Replace #ReplaceWithYourVaultName with the Backup Vault Name
\n
• Replace #ReplaceWithYourRG with the Resource Group of the Backup Vault
\n

In addition, a few notes about the script;

\n
• We are only querying for Storage Accounts in the Australia East region. You could have a single automation account querying all regions, but I prefer this way to distribute the load
\n
• We are checking if the backup already exists for the retrieved Storage Accounts and if does, we do not do anything
\n
• We have defined the Tag name on the Storage Accounts to be Environment and the value to be either Production or Non-Production
\n

Finally, as a last step, you can create the required tag in one of your Storage Accounts and run the runbook.

Once you are satisfied with the results, you can create a schedule for your Runbook and tag other Storage Accounts to automatically enable backup for Block Blobs.

Conclusion

In conclusion, automating the backup process of your Block Blob Storage is a crucial step in ensuring the safety and security of your data in case of unexpected data loss. By following the steps outlined in this post, you can set up an Azure Backup Vault to continuously back up your Block Blob Storage, without any traffic flowing between the Storage Account and the Backup Vault.

Additionally, creating an Automation Account, assigning the Backup Contributor role, and creating Backup policies can help simplify the backup process. Once you have completed these steps, you can easily run the Runbook to enable automatic backup for Block Blobs in your Storage Accounts.

Overall, this solution is a reliable and efficient way to ensure the safety of your data in the cloud.

I hope this was informative to you and thanks for reading!

Disclaimer

The sample scripts are not supported under any Microsoft standard support program or service. The sample scripts are provided AS IS without warranty of any kind. Microsoft further disclaims all implied warranties including, without limitation, any implied warranties of merchantability or of fitness for a particular purpose. The entire risk arising out of the use or performance of the sample scripts and documentation remains with you. In no event shall Microsoft, its authors, or anyone else involved in the creation, production, or delivery of the scripts be liable for any damages whatsoever (including, without limitation, damages for loss of business profits, business interruption, loss of business information, or other pecuniary loss) arising out of the use of or inability to use the sample scripts or documentation, even if Microsoft has been advised of the possibility of such damages.

Automate the backup of your block blobs with Azure Tags and Azure Automation Account.

There are a few advantages:

- You are using a managed service instead of relying on azcopy

- The restore process is a lot easier

- You can use point-in-time restore with different versions of the blob you want to restore in case it gets deleted or corrupted

- You can set different retention times based on your requirements.

What is the advantage to putting this in a backup vault vs using azcopy to copy Containers to a cool/cold/archive Container in another Storage account?