Hello howdy Readers! Thank you for visiting my post!
A lot of my customers have a business requirement that -
- once a document is written in storage account, nobody shall be able to modify or delete it, including the administrators
AND / OR - audit data can be written only once, and nobody shall be able to alter them.
These kind of business requirements are called WORM (Write Once, Read Many) and are common for various industries such as ISVs, financial or healthcare. Have you run into these mission-critical requirements? If yes, then immutable storage would be the solution.
Dependencies
The dependencies include:
- Versioning must be enabled on the storage account.
- Account kind of GPv2 (no Premium block blob support).
- ADLS Gen2 enabled accounts are not supported.
- All access tiers and redundancies are supported. If GRS, GZRS, or RA-GRS is chosen, customer-initiated failover is not supported.
Overview for Immutable Storage
The Immutable Storage for Azure Blobs feature, with the Policy Lock option, is designed to meet securities industry requirements for preserving records in a non-rewriteable and non-erasable format. In simple terms, Azure Storage is supporting WORM by default, allowing us to have data stored inside Azure Blobs that is immutable as long as we want. Even the Administrator of the Storage Account is not allowed to delete or modify the content if the WORM is active.
Azure Storage Blobs support two types of WORMS:
- Time-based Retention
- Legal Holds
The two types of WORMS are supported at account or container level. Once the policy is applied at container level all blobs under the container will respect that specific policy, the existing or new one. This feature is supported on all types of blobs, but it is recommended to use only blob types. For appearing blob and page blob, once you apply the policy, they cannot be modified anymore. Beside this, initially, they are created outside the policy and copied later inside the policy.
The following diagram shows how time-based retention policies and legal holds prevent write and delete operations while they are in effect.
Time based Retention Policy scope
A time-based retention policy can be configured at either of the following scopes:
- Version-level policy: A time-based retention policy can be configured to apply to a blob version for granular management of sensitive data. You can apply the policy to an individual version or configure a default policy for a storage account or individual container that will apply by default to all blobs uploaded to that account or container.
- Container-level policy: A time-based retention policy that is configured at the container level applies to all objects in that container. Individual objects can't be configured with their own immutability policies.
Legal hold Policy Scope
A legal hold policy can be configured at either of the following scopes:
- Version-level policy: A legal hold can be configured on an individual blob version level for granular management of sensitive data.
- Container-level policy: A legal hold that is configured at the container level applies to all blobs in that container. Individual blobs can't be configured with their own immutability policies.
How Immutable Storage works
Time Based Retention for New container
For a new container, first, Versioning needs to be enabled on the account. Second, when a container is created, VLW must be enabled with an optional time-based policy. This policy can be locked or unlocked. An unlocked default policy can be modified or deleted in the future. A locked policy cannot be deleted or reduced but it can be lengthened. Once a container has VLW enabled, it cannot be disabled though the policy if unlocked can be removed.
When a blob is written (PutBlob, PutBlockList) optionally set the VLW policy to:
- An unlocked VLW policy (even if locked VLW policy on container) with a specific retention
- A locked VLW policy with a specific retention
- No VLW policy set (disallow the container default policy)
If none of the above are chosen then, the blob is written with the default folder policy (it’s set on the blob). This implies that should the default container policy be changed (only lengthened if locked) that change will not modify the policy that exists on existing objects.
Upgrading Existing container to VLW
To upgrade an existing container with objects, you first must enable with container level WORM (CLW) with a time-based policy. Then, you can migrate to VLW. Limitations to this include:
- An existing container with legal holds in place for objects cannot be upgraded to VLW.
- An existing container that has no CLW defined with or without objects cannot be enabled for VLW.
When upgrading a CLW locked or unlocked policy to VLW, the upgrade process backfills all objects with a VersionID prior to VLW being fully enabled. The time it takes to upgrade CLW to VLW is dependent on the # of objects in the container. During the migration, the existing CLW protection is in place and, we also prevent changes to the CLW policy or container deletion (if 1+ blobs are present). During the migration, we allow new blobs to be written, reads, subject to existing CLW policy. We disallow updates to the CLW policy.
Documentation that shows on how to invoke a migration of existing container to support VLW using PowerShell.
Legal holds
If a VLW policy is set on the container, then Legal holds are not allowed on the container. Specific legal hold behavior is listed in the following sections.
Container blob policy behavior
|
Scenario |
Blob protection |
Operations Denied |
Container & Account |
|
Container Level WORM enabled container |
|||
|
Blob protected by active policy not expired or legal hold present (CLW) |
Immutable for content and user meta data |
Put (overwrite), Delete, Set Blob Metadata, Put Page, Set Blob Properties, Snapshot Blob, Incremental Copy Blob, Append Block (if allowProtectedAppendWrites not set) |
Container delete denied. Account delete denied (as long as 1+ blobs present) |
|
Blob policy expired and no legal hold present (CLW) |
Delete operations allowed. Overwrite not allowed |
Put (overwrite), Set Blob Metadata, Put Page, Set Blob Properties, Snapshot Blob, Incremental Copy Blob, Append Block (if allowProtectedAppendWrites not set) |
Container delete denied if 1+ blobs present. Storage account delete denied if locked time-based policy |
|
|
|
|
|
Version Level WORM (VLW) enabled container |
|||
|
Blob protected by an active VLW policy not expired |
Immutable for content and user meta data
|
PutBlob on existing PageBlob|AppendBlob
Cannot set legal hold on container Note: < add on versioned> |
Same as CLW behavior. If 1+ blob present OR 1+ containers present, disallow account delete
|
|
Blob policy expired |
Can delete. Overwrite allowed as it creates a new Version |
|
|
Container policy behavior
|
Scenario |
Result |
Notes |
|
Write blob |
Apply default VLW policy |
Protection based on default container policy (if default policy in place) |
|
Write blob, specify no VLW policy |
Blob with no VLW policy retention |
Blob enabled for VLW but no protection |
|
Write blob, specify locked VLW policy with specific retention |
Blob with locked VLW retention |
Blob protected. Retention date can be lengthened (but not in the past) |
|
Write blob, specify unlocked VLW policy with specific retention |
Blob with unlocked VLW retention |
Blob protected. Retention date can be lengthened or reduced |
|
Existing blob with no policy set |
Can add VLW locked or unlocked policy & retention
Can set legal hold |
|
|
Existing blob with unlocked policy set |
Can change retention (increase or decrease)
Can set legal hold |
|
|
Existing blob with locked policy set |
Can increase existing retention
Can set legal hold |
|
Note: blob policy changes are not audited. This was done to allow an infinite number of changes to the policy at the blob level. Container policy changes are consistent with CLW (limited to 5 changes through time).
Container policy behavior
|
Container State |
Allowed |
Disallowed |
Notes |
|
No Policy |
Can add CLW policy (locked or unlocked) |
|
Does not require Versioning to be enabled |
|
VLW Locked policy <new vs upgrade> |
Can change retention days (extend but not reduce) |
Cannot unlock Cannot disable versioning Cannot disable VLW Cannot add CLW |
Requires versioning |
|
VLW Unlocked policy |
Can change retention days (reduce or lengthen) Can lock policy Can remove policy
|
Can delete policy Legal holds disallowed Cannot disable versioning Cannot disable VLW Cannot add CLW |
|
|
CLW locked or unlocked policy |
Can migrate to VLW policy (requires Versioning) |
CLW behavior |
|
Configuring Version level Immutability
The SDK libraries that support immutable storage are Java, Python, Node.JS, .NET and Go. It can be also configured from PowerShell or Azure Portal.
From Azure Portal, configuring a version-level immutability policy is a two-step process:
- First, enable support for version-level immutability on a new storage account or on a new or existing container.
- Next, configure a time-based retention policy or legal hold that applies to one or more blob versions in that container.
Step 1 - To enable support for version-level immutability on a new Storage account, follow these steps:
Navigate to the Storage accounts -> On the Data protection tab, under Access control, select Enable version-level immutability support. When you check this box, the box for Enable versioning for blobs is also automatically checked.
Step 2 – To configure a time-based retention policy on a Storage account, follow these steps:
Navigate to your storage account -> Under Data management, select Data protection -> On the Data protection page, locate the Access control section. If the storage account was created with support for version-level immutability, then the Manage policy button appears in the Access control section.
Select the Manage policy button to display the Manage version-level immutability policy dialog ->Add a default time-based retention policy for the storage account.
Step 1 - To enable support for version-level immutability on a new Container, follow these steps:
Navigate to the Containers for your storage account, and select Add ->In the New container dialog, provide a name for your container, then expand the Advanced section -> Select Enable version-level immutability support to enable version-level immutability for the container.
Step 2a - To configure a time-based retention policy on a Container, follow these steps:
Navigate to the Containers and locate the container to which you want to apply the policy -> Select the More button to the right of the container name and choose Access policy -> In the Access policy dialog, under the Immutable blob storage section, choose Add policy Select Time-based retention policy and specify the retention interval.
Step 2b - To configure a legal hold on a blob version in that container, follow these steps:
Locate the target version, which may be the current version or a previous version of a blob. Select the More button and choose Access policy -> Under the Immutable blob versions section, select Add policy -> Choose Legal hold as the policy type, and select OK to apply it.
The following image shows a current version of a blob with both a time-based retention policy and legal hold configured.
If you don’t see the option available, it means that your Azure Storage account needs to be upgraded to V2. This can be done from the Azure Portal. In the Configuration Section, you will find the option to upgrade it.
Additional Information
Container and Storage Account Deletion
The deletion procedure will fail if the blob is under a WORM policy. An interesting behavior is during the deletion. You can delete a container that has immutable policies only when you don’t have any blobs inside it. Such a container can be deleted if it has 0 blobs. The same rule applies to Storage Account that can be deleted only if no active WORM policies are active or no blobs exist.
Storage Account Tiering change
This capability is available to change the tier of data even if for immutable blobs. It is allowing us to optimize cost for long-running WORM policies without affecting the policies.
Audit Logs
Both immutable policies have support for audit. Any changes that are done to the policies are automatically audited. A nice thing related to it is the period for how long this information is kept and is equal to the lifetime of the container. If you keep the container for 400 years, the audit logs are kept for 400 years.
Another feature is called Change feed support in Blob storage (details can be found here). This feature is well suited for auditing as it guarantees that all creation, modification, and deletion of files within a storage account are captured and retained. Additionally, one can enable an immutable policy that allows append blob on the $blobchangefeed container which disallows any deletion until a defined time has elapsed.
Cost
This feature is free, there is no additional cost if you activate this feature. Don’t forget that this new WORM policy can be applied on existing storage account too.
Limitations
The retention policy can be specified between 1 day to 400 years. Keep in mind that once you create a policy and set to lock state, you cannot modify it. It means that you will not be able to delete or modify content if the policy is active. Be careful during the test and integration phases.
At Storage Accounts, there are some limitations related to the number of immutable policies that can be defined. We can have a maximum of 1000 containers with immutable holds policies at each time (1000 with time-based retention, 1000 with Legal Holds).
An important thing that we need to consider is the maximum number of holds (tag holds) that we can have on the container — 10. For time-based retention policies, 1 per container, and each of them supports maximum three-time extensions.
Summary
In this blog post, I have talked about immutable storage and its limitations. If you are looking for immutable vaults for Azure Backup, it recently went generally available Immutable vaults for Azure Backup | Azure updates | Microsoft Azure. Azure Backup provides you an option to ensure that recovery points that are once created cannot be deleted before their intended expiry time. Azure Backup does this by preventing any operations which could lead to loss of backup data. Hence, this feature helps you protect your backups against threats like ransomware attacks and malicious actors by disallowing operations such as deleting backups or reducing retention in backup policies. I suggest you read about it, or I will discuss this in a future post.