Immutable Blobs Inside Azure Storage (WORM)
Published Jul 03 2023 09:52 AM 15.1K Views
Microsoft

 

Hello howdy Readers! Thank you for visiting my post! 

 

A lot of my customers have a business requirement that -

  • once a document is written in storage account, nobody shall be able to modify or delete it, including the administrators
    AND / OR
  • audit data can be written only once, and nobody shall be able to alter them.

These kind of business requirements are called WORM (Write Once, Read Many) and are common for various industries such as ISVs, financial or healthcare. Have you run into these mission-critical requirements? If yes, then immutable storage would be the solution.

 

Dependencies

The dependencies include:

  • Versioning must be enabled on the storage account.
  • Account kind of GPv2 (no Premium block blob support).
  • ADLS Gen2 enabled accounts are not supported.
  • All access tiers and redundancies are supported.  If GRS, GZRS, or RA-GRS is chosen, customer-initiated failover is not supported. 

 

Overview for Immutable Storage

The Immutable Storage for Azure Blobs feature, with the Policy Lock option, is designed to meet securities industry requirements for preserving records in a non-rewriteable and non-erasable format. In simple terms, Azure Storage is supporting WORM by default, allowing us to have data stored inside Azure Blobs that is immutable as long as we want. Even the Administrator of the Storage Account is not allowed to delete or modify the content if the WORM is active.

Azure Storage Blobs support two types of WORMS:

  1. Time-based Retention
  2. Legal Holds

The two types of WORMS are supported at account or container level. Once the policy is applied at container level all blobs under the container will respect that specific policy, the existing or new one. This feature is supported on all types of blobs, but it is recommended to use only blob types. For appearing blob and page blob, once you apply the policy, they cannot be modified anymore. Beside this, initially, they are created outside the policy and copied later inside the policy.

 

The following diagram shows how time-based retention policies and legal holds prevent write and delete operations while they are in effect.

 

khgandhi_0-1686263557663.jpeg

 

Time based Retention Policy scope

A time-based retention policy can be configured at either of the following scopes:

  • Version-level policy: A time-based retention policy can be configured to apply to a blob version for granular management of sensitive data. You can apply the policy to an individual version or configure a default policy for a storage account or individual container that will apply by default to all blobs uploaded to that account or container.
  • Container-level policy: A time-based retention policy that is configured at the container level applies to all objects in that container. Individual objects can't be configured with their own immutability policies.

 

Legal hold Policy Scope

A legal hold policy can be configured at either of the following scopes:

  • Version-level policy: A legal hold can be configured on an individual blob version level for granular management of sensitive data.
  • Container-level policy: A legal hold that is configured at the container level applies to all blobs in that container. Individual blobs can't be configured with their own immutability policies.

 

How Immutable Storage works

 

Time Based Retention for New container

For a new container, first, Versioning needs to be enabled on the account.  Second, when a container is created, VLW must be enabled with an optional time-based policy.  This policy can be locked or unlocked.  An unlocked default policy can be modified or deleted in the future. A locked policy cannot be deleted or reduced but it can be lengthened.  Once a container has VLW enabled, it cannot be disabled though the policy if unlocked can be removed.

When a blob is written (PutBlob, PutBlockList) optionally set the VLW policy to:

  1. An unlocked VLW policy (even if locked VLW policy on container) with a specific retention
  2. A locked VLW policy with a specific retention
  3. No VLW policy set (disallow the container default policy)

If none of the above are chosen then, the blob is written with the default folder policy (it’s set on the blob).  This implies that should the default container policy be changed (only lengthened if locked) that change will not modify the policy that exists on existing objects.

 

Upgrading Existing container to VLW

To upgrade an existing container with objects, you first must enable with container level WORM (CLW) with a time-based policy.  Then, you can migrate to VLW. Limitations to this include:

  • An existing container with legal holds in place for objects cannot be upgraded to VLW.
  • An existing container that has no CLW defined with or without objects cannot be enabled for VLW.

When upgrading a CLW locked or unlocked policy to VLW, the upgrade process backfills all objects with a VersionID prior to VLW being fully enabled.  The time it takes to upgrade CLW to VLW is dependent on the # of objects in the container.  During the migration, the existing CLW protection is in place and, we also prevent changes to the CLW policy or container deletion (if 1+ blobs are present).  During the migration, we allow new blobs to be written, reads, subject to existing CLW policy. We disallow updates to the CLW policy.

Documentation that shows on how to invoke a migration of existing container to support VLW using PowerShell.

 

Legal holds

If a VLW policy is set on the container, then Legal holds are not allowed on the container.  Specific legal hold behavior is listed in the following sections.

Container blob policy behavior

Scenario

Blob protection

Operations Denied

Container & Account

Container Level WORM enabled container

Blob protected by active policy not expired or legal hold present (CLW)

Immutable for content and user meta data

Put (overwrite), Delete, Set Blob Metadata, Put Page, Set Blob Properties, Snapshot Blob, Incremental Copy Blob, Append Block (if allowProtectedAppendWrites not set)

Container delete denied. Account delete denied (as long as 1+ blobs present)

Blob policy expired and no legal hold present (CLW)

Delete operations allowed.  Overwrite not allowed

Put (overwrite), Set Blob Metadata, Put Page, Set Blob Properties, Snapshot Blob, Incremental Copy Blob, Append Block (if allowProtectedAppendWrites not set)

Container delete denied if 1+ blobs present.  Storage account delete denied if locked time-based policy

 

 

 

 

Version Level WORM (VLW) enabled container

Blob protected by an active VLW policy not expired

Immutable for content and user meta data

 

 

PutBlob on existing PageBlob|AppendBlob

 

Cannot set legal hold on container

Note: < add on versioned>

Same as CLW behavior. If 1+ blob present OR 1+ containers present, disallow account delete

 

Blob policy expired

Can delete.  Overwrite allowed as it creates a new Version

 

 

Container policy behavior

Scenario

Result

Notes

Write blob

Apply default VLW policy

Protection based on default container policy (if default policy in place)

Write blob, specify no VLW policy

Blob with no VLW policy retention

Blob enabled for VLW but no protection

Write blob, specify locked VLW policy with specific retention

Blob with locked VLW retention

Blob protected.  Retention date can be lengthened (but not in the past)

Write blob, specify unlocked VLW policy with specific retention

Blob with unlocked VLW retention

Blob protected. Retention date can be lengthened or reduced

Existing blob with no policy set

Can add VLW locked or unlocked policy & retention

 

Can set legal hold

 

Existing blob with unlocked policy set

Can change retention (increase or decrease)

 

Can set legal hold

 

Existing blob with locked policy set

Can increase existing retention

 

Can set legal hold

 

 

Note: blob policy changes are not audited. This was done to allow an infinite number of changes to the policy at the blob level.   Container policy changes are consistent with CLW (limited to 5 changes through time).

Container policy behavior

Container State

Allowed

Disallowed

Notes

No Policy

Can add CLW policy (locked or unlocked)

 

Does not require Versioning to be enabled

VLW Locked policy

<new vs upgrade>

Can change retention days (extend but not reduce)

Cannot unlock

Cannot disable versioning

Cannot disable VLW

Cannot add CLW

Requires versioning

VLW Unlocked policy

Can change retention days (reduce or lengthen)

Can lock policy

Can remove policy

 

Can delete policy

Legal holds disallowed

Cannot disable versioning

Cannot disable VLW

Cannot add CLW

 

CLW locked or unlocked policy

Can migrate to VLW policy (requires Versioning)

CLW behavior

 

 

 

Configuring Version level Immutability

The SDK libraries that support immutable storage are Java, Python, Node.JS, .NET and Go. It can be also configured from PowerShell or Azure Portal.

 

From Azure Portal, configuring a version-level immutability policy is a two-step process:

  1. First, enable support for version-level immutability on a new storage account or on a new or existing container.
  2. Next, configure a time-based retention policy or legal hold that applies to one or more blob versions in that container.

Step 1 - To enable support for version-level immutability on a new Storage account, follow these steps:

Navigate to the Storage accounts -> On the Data protection tab, under Access control, select Enable version-level immutability support. When you check this box, the box for Enable versioning for blobs is also automatically checked.

khgandhi_1-1686263557667.png

 

Step 2 – To configure a time-based retention policy on a Storage account, follow these steps:

Navigate to your storage account -> Under Data management, select Data protection -> On the Data protection page, locate the Access control section. If the storage account was created with support for version-level immutability, then the Manage policy button appears in the Access control section.

 

khgandhi_2-1686263557667.png

 

Select the Manage policy button to display the Manage version-level immutability policy dialog ->Add a default time-based retention policy for the storage account.

 

khgandhi_3-1686263557668.png

 

Step 1 - To enable support for version-level immutability on a new Container, follow these steps:  

Navigate to the Containers for your storage account, and select Add ->In the New container dialog, provide a name for your container, then expand the Advanced section -> Select Enable version-level immutability support to enable version-level immutability for the container.

 

khgandhi_4-1686263557669.png

 

Step 2a - To configure a time-based retention policy on a Container, follow these steps:

Navigate to the Containers and locate the container to which you want to apply the policy -> Select the More button to the right of the container name and choose Access policy -> In the Access policy dialog, under the Immutable blob storage section, choose Add policy  Select Time-based retention policy and specify the retention interval.

 

khgandhi_5-1686263557671.png

 

Step 2b - To configure a legal hold on a blob version in that container, follow these steps:

Locate the target version, which may be the current version or a previous version of a blob. Select the More button and choose Access policy -> Under the Immutable blob versions section, select Add policy -> Choose Legal hold as the policy type, and select OK to apply it.

The following image shows a current version of a blob with both a time-based retention policy and legal hold configured.

 

khgandhi_6-1686263557672.png

 

If you don’t see the option available, it means that your Azure Storage account needs to be upgraded to V2. This can be done from the Azure Portal. In the Configuration Section, you will find the option to upgrade it.

 

Additional Information

 

Container and Storage Account Deletion

The deletion procedure will fail if the blob is under a WORM policy. An interesting behavior is during the deletion. You can delete a container that has immutable policies only when you don’t have any blobs inside it. Such a container can be deleted if it has 0 blobs. The same rule applies to Storage Account that can be deleted only if no active WORM policies are active or no blobs exist.

 

Storage Account Tiering change

This capability is available to change the tier of data even if for immutable blobs. It is allowing us to optimize cost for long-running WORM policies without affecting the policies.

 

Audit Logs

Both immutable policies have support for audit. Any changes that are done to the policies are automatically audited. A nice thing related to it is the period for how long this information is kept and is equal to the lifetime of the container. If you keep the container for 400 years, the audit logs are kept for 400 years.

 

Another feature is called Change feed support in Blob storage (details can be found here). This feature is well suited for auditing as it guarantees that all creation, modification, and deletion of files within a storage account are captured and retained. Additionally, one can enable an immutable policy that allows append blob on the $blobchangefeed container which disallows any deletion until a defined time has elapsed.

 

Cost

This feature is free, there is no additional cost if you activate this feature. Don’t forget that this new WORM policy can be applied on existing storage account too.

 

Limitations

The retention policy can be specified between 1 day to 400 years. Keep in mind that once you create a policy and set to lock state, you cannot modify it. It means that you will not be able to delete or modify content if the policy is active. Be careful during the test and integration phases.

At Storage Accounts, there are some limitations related to the number of immutable policies that can be defined. We can have a maximum of 1000 containers with immutable holds policies at each time (1000 with time-based retention, 1000 with Legal Holds).

An important thing that we need to consider is the maximum number of holds (tag holds) that we can have on the container — 10. For time-based retention policies, 1 per container, and each of them supports maximum three-time extensions.

 

Summary

In this blog post, I have talked about immutable storage and its limitations. If you are looking for immutable vaults for Azure Backup, it recently went generally available Immutable vaults for Azure Backup | Azure updates | Microsoft Azure. Azure Backup provides you an option to ensure that recovery points that are once created cannot be deleted before their intended expiry time. Azure Backup does this by preventing any operations which could lead to loss of backup data. Hence, this feature helps you protect your backups against threats like ransomware attacks and malicious actors by disallowing operations such as deleting backups or reducing retention in backup policies. I suggest you read about it, or I will discuss this in a future post.

Co-Authors
Version history
Last update:
‎Jul 03 2023 09:59 AM
Updated by: