Calculating Container Level Stats in Azure Blob Storage

Background:-

 

There would be scenarios where Customers would require container level capacity in a specific storage account. Below are multiple ways using which we can achieve this

 

Calculate the size of a Blob storage container – via Blob Inventory

 

We can make use of the Blob Inventory report and Azure Synapse Workspace to generate container-level stats. This approach would be advisable if there is versioning or snapshots enabled on the  Storage  Account.  Currently, the Blob Inventory report does not include soft deleted blobs (if  Soft delete is enabled) to calculate the capacity.  For this approach, we need to first enable the Blob Inventory report on the storage account and the next step would be to generate Container capacity using Azure Synapse Workspace.

 

The Azure Storage blob inventory feature provides an overview of your containers, blobs, snapshots, and blob versions within a storage account. Use the inventory report to understand various attributes of blobs and containers such as your total data size, age, encryption status, immutability policy, legal hold, and so on. The report provides an overview of your data for business and compliance requirements.

• For more details, please visit here:  https://docs.microsoft.com/en-us/azure/storage/blobs/blob-inventory#pricing-and-billing

Step 1: - Enable Inventory Report

 

1. Sign in to the Azure portal to get started.
2. Locate your storage account and display the account overview.
3. Under Data management, select Blob inventory.
4. Select Add your first inventory rule.

The Add a rule page appears.

5. In the Add a rule page, name your new rule.
6. Choose a container.
7. Under Object type to inventory, choose whether to create a report for blobs or containers.

If you select Blob, then under Blob subtype, choose the types of blobs that you want to include in your report, and whether to include blob versions and/or snapshots in your inventory report.

    8. Select the fields that you would like to include in your report and the format of your reports. Please make sure to include Content-Length from the fields.

   9. Choose how often you want to generate reports.

 10. Optionally, add a prefix match to filter blobs in your inventory report.

 11. Select Save.

 

Inventory output

 

Each inventory rule generates a set of files in the specified inventory destination container for that rule. The inventory output is generated under the following path:   https://<accountName>.blob.core.windows.net/<inventory-destination-container>/YYYY/MM/DD/HH-MM-SS/<ruleName where:

• accountName is your Azure Blob Storage account name.
• inventory-destination-container is the destination container you specified in the inventory rule.
• YYYY/MM/DD/HH-MM-SS is the time when the inventory began to run.
• ruleName is the inventory rule name.

 

 

Step 2:- Azure Synapse Workspace

 

Create an Azure Synapse workspace where you will execute a SQL query to report the inventory results.

 

Create the SQL query

 

After you create your Azure Synapse workspace, do the following steps.

1. Navigate to https://web.azuresynapse.net.
2. Select the Develop tab on the left edge.
3. Select the large plus sign (+) to add an item.
4. Select SQL script.

 

Run the SQL query

1. Add the following SQL query in your Azure Synapse workspace to read the inventory CSV file.

For the bulk parameter, use the URL of the inventory report CSV file that you want to analyze.

SQLCopy

 

SELECT LEFT([Name], CHARINDEX('/', [Name]) - 1) AS Container,        COUNT(*) As TotalBlobCount,        SUM([Content-Length]) As TotalBlobSizeFROM OPENROWSET(    bulk '<URL to your inventory CSV file>',    format='csv', parser_version='2.0', header_row=true) AS SourceGROUP BY LEFT([Name], CHARINDEX('/', [Name]) - 1)

 

2. Name your SQL query in the properties pane on the right.
3. Publish your SQL query by pressing CTRL+S or selecting the Publish all button.
4. Select the Run button to execute the SQL query. The blob count and total size per container are reported in the Results pane.

 

 

 

Calculate the size of a Blob storage container – via PowerShell

 

This script calculates the size of a container in Azure Blob storage by totaling the size of the blobs in the container.

 

For more details, please visit here:  https://docs.microsoft.com/en-us/azure/storage/scripts/storage-blobs-container-calculate-size-powers...

 

The script that calculates container size for billing purposes, see Calculate the size of a Blob storage container for billing purposes

 

For calculating the Container capacity with Soft delete and versioning enabled  kindly  refer to the  below script:-

• CodeRepository/GetContainerUsagealongwithSoftdeleteandversion.ps1 at master · Azure-Developer-Support/CodeRepository · GitHub

Script is to calculate the capacity of a directory:-

• https://github.com/Azure-Developer-Support/CodeRepository/blob/master/Storage/Powershell/PSListBlobsAndUsedCapacity.ps1

 

Calculate the size of a Blob storage container – via Storage explorer

 

The easiest way to calculate the size of the Blob Container is using Storage Explorer. Using the below steps, you would be able to calculate the Container level capacity. Please note that this would not give accurate capacity if Soft Delete and Versioning Enabled.

 

1. Sign-in to azure storage explorer using any of the approaches mentioned here  - https://docs.microsoft.com/en-us/azure/vs-azure-tools-storage-manage-with-storage-explorer?tabs=wind...
2. Select the storage of your interest.
3. Select the container.
4. Click the “Folder Statistics” button.

 

 

Hope this helps!