Azure PaaS Blog articles

How to get blob Total Blob Count and Total Capacity with Blob Inventory

ruineiva — Mon, 23 Mar 2026 13:46:08 GMT

Approach
Introduction to the Blob Inventory Service
Steps to enable inventory report
Support Documentation
Disclaimer:

Approach

This article presents how to take advantage of the Blob Inventory service to get the total blob count and the total capacity per storage account, per container or per directory.

I will present the steps to create the blob inventory rule and how to get the needed information without having to process the blob inventory rule results just by using the prefix match field.

Additional support documentation it is presented at the end of the article.

Introduction to the Blob Inventory Service

Azure Storage blob inventory provides a list of the containers, blobs, blob versions, and snapshots in your storage account, along with their associated properties. It generates an output report in either comma-separated values (csv) or Apache Parquet format on a daily or weekly basis. You can use the report to audit retention, legal hold or encryption status of your storage account contents, or you can use it to understand the total data size, age, tier distribution, or other attributes of your data. Please find here our documentation about the blob inventory service.

On this article, I will focus on using this service to get the blob count and the capacity.

Steps to enable inventory report

Please see below how to define a blob inventory rule to get the intended information, using the Azure Portal:

Sign in to the Azure portal to get started.
Locate your storage account and display the account overview.
Under Data management, select Blob inventory.
Select Add your first inventory rule, if you do not have any rule defined, or select Add a rule, in case that you already have at least one rule defined.
Add a new inventory rule by filling in the following fields:
1. Rule name: The name of your blob inventory rule.
2. Container: Container to store the result of the blob inventory rule execution.
3. Object type to inventory: Select blob
4. Blob types:
  1. Blob Storage: Select all (Block blobs, Page blobs, Append blobs).
  2. Data Lake Storage: Select all (Block blobs, Append blobs).
5. Subtypes:
  1. Blob Storage: Select all (Include blob versions, Include snapshots, Include deleted blobs).
  2. Data Lake Storage: Select all (Include snapshots, Include deleted blobs).
6. Blob inventory fields: Please find here all custom schema fields supported for blob inventory. In this scenario, we need to select at least the following fields:
  1. Blob Storage: Name, Creation-Time, ETag, Content-Length, Snapshot, VersionId, IsCurrentVersion, Deleted, RemainingRetentionDays.
  2. Data Lake Storage: Name, Creation-Time, ETag, Content-Length, Snapshot, DeletionId, Deleted, DeletedTime, RemainingRetentionDays.
7. Inventory frequency: A blob inventory run is automatically scheduled every day when daily is chosen. Selecting weekly schedule will only trigger the inventory run on Sundays.
  1. A daily execution will return results faster.
8. Export format: The export format. Could be a csv file or a parquet file.
9. Prefix match: Filter blobs by name or first letters. To find items in a specific container, enter the name of the container followed by a forward slash, then the blob name or first letters. For example, to show all blobs starting with “a”, type: “myContainer/a”.
  1. Here is the place to add the path where to start collecting the blob information.

The step 5.9 presented above (the prefix match field) it is the main point of this article.

Considering that we have a Storage Account with a container named work, and a directory named items, inside the container named work. Please see below how to configure the prefix match field to get the needed result:

Leave it empty to get the information at the storage account level.
Add the container name in the prefix match field to get the information at the container level.
- Put prefix match = work/
Add the directory path in the prefix match field to get the information at the directory level.
- Put prefix match = work/items/

The blob inventory execution will generate a file named <ruleName>-manifest.json, please see more information about this file in the support documentation section. This file captures the rule definition provided by the user and the path to the inventory for that rule, and the information that we want without having to process the blob inventory rule files.

{ "destinationContainer" : "inventory-destination-container", "endpoint" : "https://testaccount.blob.core.windows.net", "files" : [ { "blob" : "2021/05/26/13-25-36/Rule_1/Rule_1.csv", "size" : 12710092 } ], "inventoryCompletionTime" : "2021-05-26T13:35:56Z", "inventoryStartTime" : "2021-05-26T13:25:36Z", "ruleDefinition" : { "filters" : { "blobTypes" : [ "blockBlob" ], "includeBlobVersions" : false, "includeSnapshots" : false, "prefixMatch" : [ "penner-test-container-100003" ] }, "format" : "csv", "objectType" : "blob", "schedule" : "daily", "schemaFields" : [ "Name", "Creation-Time", "BlobType", "Content-Length", "LastAccessTime", "Last-Modified", "Metadata", "AccessTier" ] }, "ruleName" : "Rule_1", "status" : "Succeeded", "summary" : { "objectCount" : 110000, "totalObjectSize" : 23789775 }, "version" : "1.0" }

The objectCount value is the total blob count, and the totalObjectSize is the total capacity in bytes.

Special notes:

A rule needs to be defined for each path (container or directory) to get the total blob count and the total capacity.
The blob inventory rule generates a CSV or Apache Parquet formatted file(s). These files should be deleted if the blob inventory rule is only to get the information presented on this article.

Support Documentation

Topic	Some highlights
Enable Azure Storage blob inventory reports	The steps to enable inventory report.
Inventory run	If you configure a rule to run daily, then it will be scheduled to run every day. If you configure a rule to run weekly, then it will be scheduled to run each week on Sunday UTC time. The time taken to generate an inventory report depends on various factors and the maximum amount of time that an inventory run can complete before it fails is six days.
Inventory output	Each inventory rule generates a set of files in the specified inventory destination container for that rule. The inventory output is generated under the following path: https://<accountName>.blob.core.windows.net/<inventory-destination-container>/YYYY/MM/DD/HH-MM-SS/<ruleName where: *accountName* is your Azure Blob Storage account name. *inventory-destination-container* is the destination container you specified in the inventory rule. *YYYY/MM/DD/HH-MM-SS* is the time when the inventory began to run. *ruleName* is the inventory rule name.
Inventory files	Each inventory run for a rule generates the following files: Inventory file: An inventory run for a rule generates a CSV or Apache Parquet formatted file. Each such file contains matched objects and their metadata. Checksum file: A checksum file contains the MD5 checksum of the contents of manifest.json file. The name of the checksum file is <ruleName>-manifest.checksum. Generation of the checksum file marks the completion of an inventory rule run Manifest file: A manifest.json file contains the details of the inventory file(s) generated for that rule. The name of the file is <ruleName>-manifest.json. This file also captures the rule definition provided by the user and the path to the inventory for that rule.
Pricing and billing	Pricing for inventory is based on the number of blobs and containers that are scanned during the billing period.
Known Issues and Limitations	This section describes limitations and known issues of the Azure Storage blob inventory feature.

Disclaimer

These steps are provided for the purpose of illustration only.
These steps and any related information are provided "as is" without warranty of any kind, either expressed or implied, including but not limited to the implied warranties of merchantability and/or fitness for a particular purpose.
We grant You a nonexclusive, royalty-free right to use and modify the Steps and to reproduce and distribute the steps, provided that. You agree:
- to not use Our name, logo, or trademarks to market Your software product in which the steps are embedded;
- to include a valid copyright notice on Your software product in which the steps are embedded; and
- to indemnify, hold harmless, and defend Us and Our suppliers from and against any claims or lawsuits, including attorneys’ fees, that arise or result from the use or distribution of steps.

Redis Keys Statistics

LuisFilipe — Mon, 02 Feb 2026 15:57:26 GMT

Redis Keys statistics including Key Time-to-Live (TTL) statistics and Key sizes are useful for troubleshooting cache usage and performance, from client side.

This article have two sections:

First one, using a script to get statistics from Keys on Redis ( 1 Bash script + 1 LUA script - getKeyStats.sh + getKeyStats.lua )
Second one, using a script to filter and list key names ( 1 Bash script that includes a LUA script - listKeys.sh )

Key Time-to-Live (TTL):

TTL may have impact on memory usage and memory available on Redis services.

Data Loss on Redis services may happened unexpectedly due to some issue on backend, but may also happen due to Memory eviction policy, or Time-to-Live (TTL) expired.
Memory eviction policy may remove some keys from Redis service, but only when used capacity (the space used by Redis keys) reach 100% on memory available.

Not having any unexpected issue on Redis backend side or not reaching the maximum memory available, the only reason for having some keys removed from cache is due to TTL value.

TTL may not be defined at all, and in that case the key remains in the cache forever (persistent)
TTL can be set while setting a new key
TTL can be set / re-set later after key creation

TTL is defined in seconds or milliseconds, or with a negative value:

-1, the key exists but has no expiration (it’s persistent); this happens when the TTL was not defined or it was removed using PERSIST command
-2, if the key does not exist.
any other value

Related commands:

SET key1 value1 EX 60 - defines TTL as 60 seconds
SET key1 value1 PX 60000 - defines TTL as 60000 milliseconds (60 seconds)
EXPIRE key1 60 - Set a timeout of 60 seconds on key1
TTL key1 - returns the current TTL value, in seconds
PTTL key1 - returns the current TTL value, in milliseconds
PERSIST key1 removes TTL from that key and make the key persistent

Notes:

TTL counts down in real time, but Redis expiration is lazy + active, so exact timing isn’t guaranteed to the millisecond.
A TTL of 0 is basically a race condition, that usually are not seen, it because the key expires immediately.
EXPIRE key 0 deletes the key right away.

There is no guarantee the deletion happens exactly at expiration time. Redis lazy + active expiration means the key is checked only when someone touches it (lazy), but to avoid memory filling up with expired junk, Redis also runs a background job to periodically scan a subset of keys and delete the expired ones (active). So, some expired keys may survive a bit longer, not accessible anymore but still im memory.

Example Redis lazy:

at 11:59:00 SET key1 value1 EX 60 - 60 seconds expiration time
key1 expires at 12:00:00
no one accesses it until 12:00:05 - when someone try to access key1 at 12:00:05, Redis identify the key1 expired and delete it.

Example Redis active:

for the same key1, after 12:00:00. if during the periodically background job Redis scan the subset of keys containing key1, that key1 will be actively deleted.

For that reason, we may see some higher memory usage than the real memory used by active keys in the cache.
For more information about Redis commands, check Redis Inc - Commands

Key Sizes:

Large key value sizes in the cache, may have high impact on Redis performance.
Redis service is designed to 1KB response size, and Microsoft recommends to use up to 100KB on Azure Redis services, to get a better performance.
Redis response size may not be exactly the same as key size, as Response size is the sum of the response from each operation sent to Redis.
While the response size can be the size of only one key requested (like GET), we can see very often response size being a sum of more than one key, as result of multikey operations (like MGET and others).

The scope of this article is the focus on each key size; so, we will not discuss on this article the implications of multikey commands.

By design Redis service is a single thread system per shard, and this is not a Microsoft/Azure limitation but a Redis design feature.
To be very quick on processing requests, Redis is optimized to work and process small keys, and for that is more efficient using a single thread instead of the need of context switching.
In a multi threaded system, context switching happens when the processor stops executing one thread and starts executing another.
When that happens, the OS saves the current thread’s state (registers, program counter, stack pointer, etc.) and restores the state of the next thread.
To save time on that process, Redis service is designed to run in a single thread system.

Due to the single thread nature, all operations sent to Redis service, are waiting in a queue to be processed.
To minimize latency, all keys must remain small so they can be processed efficiently and responses can be transmitted to the client quickly over the network.

For that reason, it's important to understand the key sizes we have on our Redis service, and maintain all keys as small as possible.

Scripts Provided

To help on identifying some specific TTL values and Keys sizes in a Redis cache, two solutions are provided below:

1. Get Key statistics - that scans all cache and return the amount of Redis keys with:

- Number of keys with TTL no set
- Number of keys with TTL higher or equal to a user defined TTL threshold
- Number of keys with TTL lower than a user defined TTL threshold
- Number of keys with value size higher or equal than a user defined Size threshold
- Number of keys with value size lower than a user defined Size threshold
- Total number of keys in the cache.
- It also includes start and end time, and the total time spent on the keys scan.

2. List Key Names - this script returns a list of Redis Keys names, based on parameters provided:

- No TTL set, or
- TTL higher or equal to a user defined TTL threshold, or
- TTL lower than to a user defined TTL threshold
- Key value size higher or equal than a user defined Size threshold, or
- Key value size lower than a user defined Size threshold
- Total number of keys in the cache
- It also includes start and end time, and the total time spent on the keys scan.

WARNING:

Due to the need to read all keys in the cache, both solutions can cause high workload on Redis side, specially for high datasets on the cache, with high number of keys.
Both solutions are using LUA script that runs on Redis side, and depending on the amount of keys in the cache, may block all other commands to be processed, while the script is running.
The duration time on the output from each script run, may help to identify the impact of the scripts to run.
Run it carefully and do some tests first on your developing environment, before using in a production.

YOU CAN RUN THE BELOW SCRIPTS AT YOUR OWN RISK.
WE DON'T ASSUME ANY RESPONSABILITY FOR UNEXPECTED RESULTS.

1- Get Key statistics

To get Redis key statistics, we use Linux Bash shell and Redis-cli tool to run LUA script on Redis side, to get TTL values and sizes from each key.
This solution is very fast, but needs to scan all keys in the cache during the LUA script run.
Despite very quick, the time depends of the amount of keys to scan.
This may block Redis to process other requests, due to the single-thread nature of Redis service.

The below script scans all cache and return only the amount of Redis keys with:

- Number of keys with TTL no set
- Number of keys with TTL higher or equal to a user defined TTL threshold
- Number of keys with TTL lower than a user defined TTL threshold
- Number of keys with value size higher or equal than a user defined Size threshold
- Number of keys with value size lower than a user defined Size threshold
- Total number of keys in the cache.
- It also includes start and end time, and the total time spent on the keys scan.

Calling getKeyStats.sh return statistics from the existing keys in the cache, based on the two threshold values (optional), that can be passed on script command line parameters.

This script can clarify questions like:

“Do I have any key in the cache without TTL set?”, or
“Why my keys are not expiring?” (any threshold values used will clarify this)
“Do I have large keys in my cache, larger than 1KB (any TTL threshold can be used, with key size threshold 1024)
“How many keys I have that will expire on next 1 hour?” (threshold 3600, with any key size threshold)

Output:

How to run:

create the below getKeyStats.sh and getKeyStats.lua files on same folder, on your Linux environment (Ubuntu 20.04.6 LTS used)
give permissions to run Shell script, with command chmod 700 getKeyStats.sh
Call the script using the syntax:

./getKeyStats.sh host password [port] [ttl_threshold] [size_threshold]

Script parameters:

host (mandatory) : the URI for the cache
password (mandatory) : the Redis access key from the cache
port (optional - default 10000) : TCP port used to access the cache
ttl_threshold (optional - default 600 - 10 minutes) : Key TTL threshold (in seconds) to be used on the results (use -1 to 1 to get Keys with no TTL set)
size_threshold (optional - default 102400 - 100KB) : Key Size threshold to be used on the results

If not provided, the default values will be used: Redis Port: 10000, ttl_threshold: 600 Seconds, size_threshold: 102400 Bytes (100KB).

Tested with:

Ubuntu 20.04.6 LTS
redis-cli -v
redis-cli 7.4.2
Redis services:
- Azure Managed Redis Balanced B0 OSSMode
- Azure Cache for Redis Standard C1

getKeyStats.sh

#!/usr/bin/env bash #============================== LUA script version ================= # Linux Bash Script to get statistics from Redis Keys TTL values and Key value sizes # It returns the Number of: # - keys with TTL no set # - keys with TTL higher or equal to TTL_treshold # - keys with TTL lower TTL_threshold # - keys with value size higher or equal than Size_threshold # - keys with value size lower than Size_threshold # - total number of keys in the cache. #------------------------------------------------------- # WARNING: # It uses LUA script to run on Redis server side. # Use it carefully, during low Redis workoads. # Do your tests first on a Dev environment, before use it on production. #------------------------------------------------------- # It requires : # redis-cli v7 or above #-------------------------------------------------------- # Usage: # getRedisTTL.sh <cacheuri> <cacheaccesskey> [<accessport>(10000)] [<ttl_treashold>(600)] [<size_threshold>(102400)] #======================================================== #------------------------------------------------------ # To use non-ssl port requites to remove --tls parameter from Redis-cli command below #------------------------------------------------------ # Parameters REDIS_HOST="${1:?Usage: $0 <host> <password> [port] [ttl_threshold] [Size_Threshold]}" REDISCLI_AUTH="${2:?Usage: $0 <host> <password> [port] [ttl_threshold] [Size_Threshold]}" REDIS_PORT="${3:-10000}" # 10000 / 6380 / 6379 REDIS_TTL_THRESHOLD="${4:-600}" # 10 minutes REDIS_SIZE_THRESHOLD="${5:-102400}" # 100KB # Port number must be numeric if ! [[ "$REDIS_PORT" =~ ^[0-9]+$ ]]; then echo "ERROR: Redis Port must be numeric" exit 1 fi # TTL threshold must be numeric if ! [[ "$REDIS_TTL_THRESHOLD" =~ ^[0-9]+$ ]]; then echo "ERROR: TTL threshold must be numeric" exit 1 fi # Size threshold must be numeric if ! [[ "$REDIS_SIZE_THRESHOLD" =~ ^[0-9]+$ ]]; then echo "ERROR: Size threshold must be numeric" exit 1 fi echo "" echo "========================================================" echo "Scaning number of keys with TTL threshold $REDIS_TTL_THRESHOLD Seconds, and Key size threshold $REDIS_SIZE_THRESHOLD Bytes" # Start time start_ts=$(date +%s.%3N) echo "Start time: $(date "+%d-%m-%Y %H:%M:%S")" echo "------------------------" echo "" # Procesing result=$(redis-cli \ -h "$REDIS_HOST" \ -a "$REDISCLI_AUTH" \ -p "$REDIS_PORT" \ --tls \ --no-auth-warning \ --raw \ --eval getKeyStats.lua , "$REDIS_TTL_THRESHOLD" "$REDIS_SIZE_THRESHOLD" \ | tr '\n' ' ') read no_ttl nonexist ttl_high ttl_low ttl_invalid size_high size_low size_nil total <<< "$result" if [[ $result == ERR* ]]; then echo "Redis Lua error:" echo "$result" else echo "Total keys scanned: $total" echo "------------" echo "Keys with TTL not set : $no_ttl" echo "Keys with TTL >= $REDIS_TTL_THRESHOLD seconds: $ttl_high" echo "Keys with TTL < $REDIS_TTL_THRESHOLD seconds: $ttl_low" echo "Keys with TTL invalid/error : $ttl_invalid" echo "Non existent Keys : $nonexist" echo "------------" echo "Keys with Size >= $REDIS_SIZE_THRESHOLD Bytes: $size_high" echo "Keys with Size < $REDIS_SIZE_THRESHOLD Bytes: $size_low" echo "Keys with invalid Size : $size_nil" fi echo "" echo "------------------------" end_ts=$(date +%s.%3N) echo "End time: $(date "+%d-%m-%Y %H:%M:%S")" # Duration - Extract days, hours, minutes, seconds, milliseconds duration=$(awk "BEGIN {print $end_ts - $start_ts}") days=$(awk "BEGIN {print int($duration/86400)}") hours=$(awk "BEGIN {print int(($duration%86400)/3600)}") minutes=$(awk "BEGIN {print int(($duration%3600)/60)}") seconds=$(awk "BEGIN {print int($duration%60)}") milliseconds=$(awk "BEGIN {printf \"%03d\", ($duration - int($duration))*1000}") echo "Duration : ${days} days $(printf "%02d" "$hours"):$(printf "%02d" "$minutes"):$(printf "%02d" "$seconds").$milliseconds" echo "========================================================"

getKeyStats.lua

local ttl_threshold = tonumber(ARGV[1]) local size_threshold = tonumber(ARGV[2]) local cursor = "0" -- Counters local no_ttl = 0 local nonexist = 0 local ttl_high = 0 local ttl_low = 0 local ttl_invalid = 0 local size_high = 0 local size_low = 0 local size_nil = 0 local total = 0 repeat local scan = redis.call("SCAN", cursor, "COUNT", 1000) cursor = scan[1] local keys = scan[2] for _, key in ipairs(keys) do local ttl = redis.call("TTL", key) local size = redis.call("MEMORY","USAGE", key) total = total + 1 if ttl == -1 then no_ttl = no_ttl + 1 elseif ttl == -2 then nonexist = nonexist + 1 elseif type(ttl) ~= "number" then ttl_invalid = ttl_invalid + 1 elseif ttl >= ttl_threshold then ttl_high = ttl_high + 1 else ttl_low = ttl_low + 1 end if size == nil then size_nil = size_nil + 1 elseif size >= size_threshold then size_high = size_high + 1 else size_low = size_low + 1 end end until cursor == "0" return { no_ttl, nonexist, ttl_high, ttl_low, ttl_invalid, size_high, size_low, size_nil, total }

Performance:

Redis service used: Azure Managed Redis - Balanced B0 - OSSMode

Scanning number of keys with TTL threshold 600 Seconds, and Key size threshold 102400 Bytes

Total keys scanned: 46161

TTL not set : 0
TTL >= 600 seconds: 46105
TTL < 600 seconds: 56
TTL invalid/error : 0
Non existent key : 0

Keys with Size >= 102400 Bytes: 0
Keys with Size < 102400 Bytes: 46161
Keys with invalid Size : 0

Duration : 0 days 00:00:00.602
# ------------------

Redis service used: Azure Cache for Redis - Standard - C1

Scanning number of keys with TTL threshold 100 Seconds, and Key size threshold 500 Bytes

Total keys scanned: 1227

TTL not set : 2
TTL >= 100 seconds: 1225
TTL < 100 seconds: 0
TTL invalid/error : 0
Non existent key : 0

Keys with Size >= 500 Bytes: 1225
Keys with Size < 500 Bytes: 2
Keys with invalid Size : 0

Duration : 0 days 00:00:00.630
# ------------------

WARNING:
The above scripts uses LUA script, that runs on Redis side, and may block you normal workload.
Use it carefully when have a large number of keys in the cache, and during low workload times.

2 - List Key Names

Once we identify some amount of keys in the cache with some specific threshold, we may want to list that key names.
The below script can help on that, and returns a list of Redis Keys names with:

No TTL set
TTL higher or equal to a user defined TTL threshold
TTL lower than to a user defined TTL threshold
Key value size higher or equal than a user defined Size threshold
Key value size lower than a user defined Size threshold
Total number of keys in the cache
It also includes start and end time, and the total time spent on the keys scan.

Calling getKeyStats.sh return just key names and respective TTL values and key sizes.
The result depends of the threshold values used, that can be passed on script command line parameters or using the default ones.
On this script, we can use a sign on both threshold values:

“-“, if we want to return keys lower than that threshold;
“+” or no sign, if we want to return keys higher than that threshold.

This script can clarify questions like:

“What are the keys I have in the cache without TTL?” (Threshold values: -1 0)
“What are the keys I have in the cache with TTL, and size higher than 1KB”? (Threshold values: 0 1024)
“What are the keys I have in the cache with TTL, and size smaller than 1KB ?” (Threshold values: 0 -1024)
“What are the keys I have in the cache with that will expire on next 1 hour?” (Threshold values: -3600 0)
“What are the keys I have in the cache with that will expire after 1 hour?” (Threshold values: +3600 0)
“What are the keys I have in the cache with No TTL set and size larger than 200K?” (Threshold values: -1 204800)

Output:

How to run:

create the below listKeys.sh file under some folder, on your Linux environment (Ubuntu 20.04.6 LTS used)
give permissions to run Shell script, with command chmod 700 listKeys.sh
Call the script using the syntax:

./listKeys.sh host password [port] [+/-][ttl_threshold] [+/-][size_threshold]

Script parameters:

host (mandatory) : the URI for the cache
password (mandatory) : the Redis access key from the cache
port (optional - default 10000) : TCP port used to access the cache
[+/-] (optional) before ttl_threshold: indicates if we want return keys with lower "-", or higher TTL "+" than ttl_threshold
ttl_threshold (optional - default 600 - 10 minutes) : Key TTL threshold (in seconds) to be used on the results (use -1 to get Keys with no TTL set)
[+/-] (optional) before size_threshold: indicates if we want return keys with small size "-", or large size "+" than size_threshold
size_threshold (optional - default 102400 - 100KB) : Key Size threshold to be used on the results

If not provided, the default values that will be used: No TTL set (-1), and Key size threshold 102400 Bytes (100KB).

Tips:

use ttl_threshold = -1 to return key names with no TTL (ex: /listKeys.sh [port] -1 [+/-][size_Threshold])
use ttl_threshold = -500 to return key names with TTL below 500 seconds (ex: /listKeys.sh [port] -500 [+/-][size_Threshold])
use ttl_threshold = 500 to return key names with TTL above or equal to 500 seconds (ex: /listKeys.sh [port] 500 [+/-][size_Threshold])

use size_threshold = 0 to return key names with any size in the cache (ex: /listKeys.sh [port] [+/-][ttl_threshold] 0)
use size_threshold = -1000 to return key names with size below 1000 Bytes (ex: /listKeys.sh [port] [+/-][ttl_threshold] -1000)
use size_threshold = 1000 to return key names with size above or equal to 1000 Bytes (ex: /listKeys.sh [port] [+/-][ttl_threshold] 1000)

use ttl_threshold = 0 AND size_threshold = 0 to return all key names with any TTL and any size in the cache (ex: /listKeys.sh [port] 0 0)
use ttl_threshold = -1 AND size_threshold = 0 to return all key names with no TTL and any size in the cache (ex: /listKeys.sh [port] -1 0)

Tested with:

Ubuntu 20.04.6 LTS
redis-cli -v
redis-cli 7.4.2
Redis services:
- Azure Managed Redis Balanced B0 OSSMode
- Azure Cache for Redis Standard C1

listKeys.sh

#!/usr/bin/env bash set -euo pipefail #============================== LUA script version ================= # Linux Bash Script to list Redis Keys names # It returns key names with: # - No TTL set # - with TTL higher or equal to TTL_treshold # - with TTL lower TTL_threshold # - with value size higher or equal than Size_threshold # - with value size lower than Size_threshold # - total number of keys in the cache. #------------------------------------------------------- # WARNING: # It uses LUA script (included on Bash code) to run on Redis server side. # Use it carefully, during low Redis workoads. # Do your tests first on a Dev environment, before use it on production. #------------------------------------------------------- # It requires : # redis-cli v7 or above #-------------------------------------------------------- # Usage: # listKeys.sh <cacheuri> <cacheaccesskey> [<accessport>(10000)] [+/-][<ttl_treashold>(-1)] [+/-][<size_treashold>(102400)] #======================================================== #------------------------------------------------------ # Using non-ssl port requires to remove --tls parameter on Redis-cli command below #------------------------------------------------------ sintax="<redis_host> <password> [redis_port] [+/-][ttl_threshold] [+/-][size_threshold]" REDIS_HOST="${1:?Usage: $0 $sintax}" REDISCLI_AUTH="${2:?Usage: $0 $sintax}" REDIS_PORT="${3:-10000}" # Redis port (10000, 6380, 6379) KEYTTL_THRESHOLD=${4:-"-1"} # -1, +TTL_threshold, TTL_threashold, -TTL_threshold KEYSIZE_THRESHOLD="${5:-102400}" # +Size_threshold, Size_threashold, -Size_threshold # Port number must be numeric if ! [[ "$REDIS_PORT" =~ ^[0-9]+$ ]]; then echo "ERROR: Redis Port must be numeric" exit 1 fi # Check if KEYTTL_THRESHOLD is a valid integer if ! [[ "$KEYTTL_THRESHOLD" =~ ^[-+]?[0-9]+$ ]]; then echo "Error: ttl_threshold $KEYTTL_THRESHOLD is not an integer" exit 1 fi # Check if KEYSIZE_THRESHOLD is a valid integer if ! [[ "$KEYSIZE_THRESHOLD" =~ ^[-+]?[0-9]+$ ]]; then echo "Error: Size_threshold $KEYSIZE_THRESHOLD is not an integer" exit 1 fi # Check if TTL Threasold is positive (or zero), or negative if [ "$KEYTTL_THRESHOLD" -ge 0 ]; then TTLSIGN="+" else TTLSIGN="-" fi # Check if Size Threshold is positive (or zero), or negative if [ "$KEYSIZE_THRESHOLD" -ge 0 ]; then SIZESIGN="+" size_text="larger than" else SIZESIGN="-" size_text="smaler than" fi # specific with no TTL set if [ "$KEYTTL_THRESHOLD" -eq -1 ]; then ttl_text="No TTL set" fi if [ "$KEYTTL_THRESHOLD" -ge 0 ]; then ttl_text="TTL above $KEYTTL_THRESHOLD Seconds" fi if [ "$KEYTTL_THRESHOLD" -lt -1 ]; then ttl_text="TTL below ${KEYTTL_THRESHOLD#[-+]} Seconds" fi # remove any sign KEYTTL_THRESHOLD="${KEYTTL_THRESHOLD#[-+]}" KEYSIZE_THRESHOLD="${KEYSIZE_THRESHOLD#[-+]}" echo "========================================================" echo "List all key names with $ttl_text, and Key size $size_text $KEYSIZE_THRESHOLD Bytes" # Start time start_ts=$(date +%s.%3N) echo "Start time: $(date "+%d-%m-%Y %H:%M:%S")" echo "------------------------" echo "" # Procesing redis-cli -h "$REDIS_HOST" -p "$REDIS_PORT" -a "$REDISCLI_AUTH" --tls --no-auth-warning EVAL " local cursor = '0' local ttl_threshold = tonumber(ARGV[1]) -- KEYTTL_THRESHOLD local ttl_sign = ARGV[2] -- TTLSIGN local size_threshold = tonumber(ARGV[3]) -- KEYSIZE_THRESHOLD local size_sign = ARGV[4] -- SIZESIGN local output = {} local count = 0 local totalKeys = 0 local strKeyTTL = '' local strKeySize = '' -- Scanning keys in the cache table.insert(output, '--------------------------------------') repeat local res = redis.call('SCAN', cursor, 'COUNT', 100) cursor = res[1] for _, k in ipairs(res[2]) do local ttl = redis.call('TTL', k) local size = redis.call('MEMORY','USAGE', k) totalKeys = totalKeys + 1 if (size_sign == '+' and size >= size_threshold) or (size_sign == '-' and size < size_threshold) then -- TTL == -1 → no expiration if ttl_sign == '-' and ttl_threshold == 1 then if ttl == -1 then table.insert(output, k .. ': TTL: -1, Size: ' .. size .. ' Bytes') count = count + 1 end -- TTL comparisons (exclude -1 and -2) else if ttl >= 0 then if ttl_sign == '-' and ttl < ttl_threshold then table.insert(output, k .. ': TTL: ' .. ttl .. ' seconds, Size: ' .. size .. ' Bytes') count = count + 1 elseif ttl_sign == '+' and ttl >= ttl_threshold then table.insert(output, k .. ': TTL: ' .. ttl .. ' seconds, Size: ' .. size .. ' Bytes') count = count + 1 end end end end end until cursor == '0' -- Adding summary to output table.insert(output, '--------------------------------------') if (size_sign == '+') then strKeySize = 'larger' else strKeySize = 'smaler' end strKeySize = 'size ' .. strKeySize .. ' than ' .. size_threshold .. ' Bytes' if ttl_sign == '-' and ttl_threshold == 1 then strKeyTTL = 'No TTL' elseif ttl_sign == '-' then strKeyTTL = 'TTL < ' .. ttl_threshold .. ' seconds' elseif ttl_sign == '+' then strKeyTTL = 'TTL >= ' .. ttl_threshold .. ' seconds' end strKeyTTL = ' keys found with ' .. strKeyTTL table.insert(output, 'Scan completed.') table.insert(output, 'Total of ' .. totalKeys .. ' keys scanned.') table.insert(output, count .. strKeyTTL .. ', and ' .. strKeySize) table.insert(output, '--------------------------------------') return output " 0 "$KEYTTL_THRESHOLD" "$TTLSIGN" "$KEYSIZE_THRESHOLD" "$SIZESIGN" echo " " end_ts=$(date +%s.%3N) echo "End time: $(date "+%d-%m-%Y %H:%M:%S")" # Duration - Extract days, hours, minutes, seconds, milliseconds duration=$(awk "BEGIN {print $end_ts - $start_ts}") days=$(awk "BEGIN {print int($duration/86400)}") hours=$(awk "BEGIN {print int(($duration%86400)/3600)}") minutes=$(awk "BEGIN {print int(($duration%3600)/60)}") seconds=$(awk "BEGIN {print int($duration%60)}") milliseconds=$(awk "BEGIN {printf \"%03d\", ($duration - int($duration))*1000}") echo "Duration : ${days} days $(printf "%02d" "$hours"):$(printf "%02d" "$minutes"):$(printf "%02d" "$seconds").$milliseconds" echo "========================================================"

Performance:

Redis service used: Azure Managed Redis Balanced B0 OSSMode

# ------------------
Scan completed. Total keys listed: 46005
Duration : 0 days 00:00:01.437

# ------------------
Redis service used: Azure Cache for Redis - Standard - C1
Scan completed. Total keys listed: 1225
Duration : 0 days 00:00:00.545

# ------------------

WARNING:
The above script uses LUA script, that runs on Redis side, and may block you normal workload.
Use it carefully when have a large number of keys in the cache, and during low workload times.

YOU CAN RUN THE BELOW SCRIPTS AT YOUR OWN RISK.
WE DON'T ASSUME ANY RESPONSABILITY FOR UNEXPECTED RESULTS.

References

I hope this can be useful !!!

Deriving expiry days and remaining retention days for blobs through blob inventory

Harshi_mrinal — Tue, 11 Nov 2025 05:35:25 GMT

In managing data within Azure blob storage accounts and Azure data lake gen 2 storage accounts, organizations often encounter scenarios where blobs have been deleted but remain in a soft-deleted state. To calculate the remaining retention days for all such blobs across an entire storage account can be a critical requirement for customers seeking to optimize data management and ensure compliance with retention policies.

Additionally, certain blobs may have an expiry time set, scheduling their deletion for a future date. To facilitate the identification and monitoring of these blobs and their respective expiry times, a custom query has been written to efficiently list and calculate expiry information, enabling users to proactively manage their storage resources.

The expiry time for Azure blobs is set using the Set Blob Expiry operation. This feature is present in only Hierarchical namespace enabled storage accounts.

We can set the expiry with below steps:

i) Azure Storage action- About Azure Storage Actions - Azure Storage Actions | Microsoft Learn
Storage action can be used to set blob expiry, share a high-level snippet for the operation below

ii) REST API- Set Blob Expiry (REST API) - Azure Storage | Microsoft Learn to set the expiry time for your blobs. This ensures that each blob has a defined lifecycle and will be deleted after the specified period.

In this blog, it is a step-by-step process of listing the expiry time and retention of the blobs using Blob Inventory report and then parsing it using Synapse.

1. Set blob inventory rule

Get CSV file from blob inventory run.
Go to the container where inventory reports are getting stored.
Navigate to the recent date folder and get url of Blob Inventory csv life.
Sharing the below snippet for reference:

2. Create an Azure Synapse workspace

Next, create an Azure Synapse workspace where you will execute a SQL query to report the inventory results.

Create the SQL query: After you create your Azure Synapse workspace, do the following steps.

Navigate to https://web.azuresynapse.net.
Select the Develop tab on the left edge.
Select the large plus sign (+) to add an item.
Select SQL script.

3. Use the sample query below to get the expiry time and remaining retention days of blob respectively

select

LEFT([Name], CHARINDEX('/', [Name]) - 1) AS Container,

RIGHT([Name], LEN([Name])- CHARINDEX('/',[Name])) AS Blob,

[Expiry-time]

from OPENROWSET(

bulk '<URL to your inventory CSV file>',

format='csv', parser_version='2.0', header_row=true

) as Source

For blobs which got deleted directly, you can calculate the remaining retention days since the data is present in soft deleted state and will be deleted permanently after the retention days complete.

select

LEFT([Name], CHARINDEX('/', [Name]) - 1) AS Container,

RIGHT([Name], LEN([Name])- CHARINDEX('/',[Name])) AS Blob,

[Expiry-time],

RemainingRetentionDays

from OPENROWSET(

bulk '<URL to your inventory CSV file>',

format='csv', parser_version='2.0', header_row=true

) as Source

In the above snippet, the Null value represents that the blob is not deleted and no expiry time is set on the blob yet.

Please Note: Calculating blob expiry from blob inventory is one way, customer can explore other options such as Powershell and Azure CLI to achieve the same.

Reference links:-
Set Blob Expiry (REST API) - Azure Storage | Microsoft Learn
Create a storage task - Azure Storage Actions | Microsoft Learn
Azure Storage blob inventory | Microsoft Learn
Calculate blob count and size using Azure Storage inventory | Microsoft Learn

Exclude Prefix in Azure Storage Action: Smarter Blob Management

ManjunathS — Tue, 30 Sep 2025 09:26:23 GMT

Azure Storage Actions is a powerful platform for automating data management tasks across Blob and Data Lake Storage. Among its many features, Exclude Prefix stands out as a subtle yet critical capability that helps fine-tune task assignments.

What Is the "Exclude Prefix" Feature?

The Exclude Prefix option allows users to omit specific blobs or folders from being targeted by Azure Storage Actions. This is particularly useful when applying actions such as:

Moving blobs to a cooler tier
Deleting blobs
Rehydrating archived blobs
Triggering workflows based on blob changes

For example, if you're running a task to archive blobs older than 30 days, but you want to exclude logs or config files, you can define a prefix like logs/ or config/ in the exclusion list.

How to Use It — Example Scenario:
In the following example, blobs across the entire storage account were deleted based on a condition: if a blob’s access tier was set to Hot, it was deleted except for those blobs or paths explicitly listed under the Exclude blob prefixes property.

Create a Task: -
- Navigate to the Azure portal and search for Storage tasks. Then, under Services, click on Storage tasks – Azure Storage Actions

- On the Azure Storage Actions | Storage Tasks page, click Create to begin configuring a new task.

- Complete all the required fields, then click Next to proceed to the Conditions page. To configure blob deletion, add the following conditions on the Conditions page.

Add the Assignment :-
- Click Add assignment in the Select scope section, choose your subscription and storage account, then provide a name for the assignment.
- In the Role assignment section, select Storage Blob Data Owner from the Role drop-down list to assign this role to the system-assigned managed identity of the storage task.
- In the Filter objects section, specify the Exclude Blob Prefix filter if you want to exclude specific blobs or folders from the task.

In the example specified above, blobs will be deleted—except for those under the path “excludefiles” listed in the Exclude blobprefixes property.

- In the Trigger details section, choose the runs of the task and then
  select the container where you'd like to store the execution reports.
- Select Add. In the Tags tab, select Next and in the Review + Create tab, select Review + create.
- When the task is deployed, your deployment is complete page appears and select Go to resource to open the Overview page of the storage task.
Enable the Task Assignment: -
- In the Trigger details section, we have a Enable task assignment checkbox which is checked by default.

- If the Enable task assignments checkbox is unchecked, you can still enable assignments manually from the Assignments page. To do this, go to Assignments, select the relevant assignment, and then click Enable.

The task assignment is queued to run and will run at the specified time.

Monitoring the runs:-
After the task completes running, you can view the results of the run.
- With the Assignments page still open, select View task runs.
- Select the View report link to download a report. You can also view these comma-separated reports in the container that you specified when you configured the assignment.

Conclusion:
The Exclude Prefix feature in Azure Storage Actions provides enhanced control and flexibility when managing blob data at scale. By allowing you to exclude specific prefixes from actions like delete or tier changes, it helps you safeguard critical data, reduce mistakes, and fine-tune automation workflows. This targeted approach not only improves operational efficiency but also supports more granular data in Azure Blob Storage.

Note:-

Azure Storage Actions are generally available in the following public regions: https://learn.microsoft.com/en-us/azure/storage-actions/overview#supported-regions
We can also exclude certain blobs using the “Not”operator when building task conditions. Blobs may be excluded based on specific blob or container attributes from the task conditions side as well—not just through task assignments.
In the screenshot below, we are using the Not operator (!) to exclude blobs where the blob name is equal to "Test".

Please refer: https://learn.microsoft.com/en-us/azure/storage-actions/storage-tasks/storage-task-conditions#multiple-clauses-in-a-condition.

Reference Links:-

Developer Tier APIM + Self-hosted Gateway

reve — Sat, 27 Sep 2025 16:45:43 GMT

Developer tier APIM enjoys many "premium" features such as vnet injection.

Feature	Consumption	Developer	Basic	Basic v2	Standard	Standard v2	Premium	Premium v2 (preview)
Microsoft Entra integration¹	No	Yes	No	Yes	Yes	Yes	Yes	Yes
Virtual network injection support	No	Yes	No	No	No	No	Yes	Yes
Private endpoint support for inbound connections	No	Yes	Yes	No	Yes	Yes	Yes	No
Outbound virtual network integration support	No	No	No	No	No	Yes	No	Yes
Multi-region deployment	No	No	No	No	No	No	Yes	No
Availability zones	No	No	No	No	No	No	Yes	No
Multiple custom domain names for gateway	No	Yes	No	No	No	No	Yes	No
Developer portal²	No	Yes	Yes	Yes	Yes	Yes	Yes	Yes
Built-in cache	No	Yes	Yes	Yes	Yes	Yes	Yes	Yes
External cache	Yes	Yes	Yes	Yes	Yes	Yes	Yes	Yes
Autoscaling	No	No	Yes	No	Yes	No	Yes	No
API analytics	No	Yes	Yes	Yes	Yes	Yes	Yes	Yes
Self-hosted gateway³	No	Yes	No	No	No	No	Yes	No
Workspaces	No	No	No	No	No	No	Yes	Yes
TLS settings	Yes	Yes	Yes	Yes	Yes	Yes	Yes	Yes
Client certificate authentication	Yes	Yes	Yes	Yes	Yes	Yes	Yes	Yes
Policies⁴	Yes	Yes	Yes	Yes	Yes	Yes	Yes	Yes
Credential manager	Yes	Yes	Yes	Yes	Yes	Yes	Yes	Yes
Backup and restore	No	Yes	Yes	No	Yes	No	Yes	No
Management over Git	No	Yes	Yes	No	Yes	No	Yes	No
Azure Monitor metrics	Yes	Yes	Yes	Yes	Yes	Yes	Yes	Yes
Azure Monitor and Log Analytics request logs	No	Yes	Yes	Yes	Yes	Yes	Yes	Yes
Application Insights request logs	Yes	Yes	Yes	Yes	Yes	Yes	Yes	Yes
Static IP	No	Yes	Yes	No	Yes	No	Yes	No
Export API to Power Platform	Yes	Yes	Yes	Yes	Yes	Yes	Yes	Yes
Export API to Postman	Yes	Yes	Yes	Yes	Yes	Yes	Yes	Yes
Export API to MCP server (preview)	No	No	Yes	Yes	Yes	Yes	Yes	Yes
Expose existing MCP server (preview)	No	No	Yes	Yes	Yes	Yes	Yes	Yes

Feature-based comparison of Azure API Management tiers | Microsoft Learn

It however comes with weaknesses like no SLA guaranteed since it's designed for non-production use cases and evaluations in the first place.

For projects that benefit from the ROI of developer tier APIM and would like to have more control over APIM service availability, developer tier + self-hosted gateway might be an option. One example use case here is to provision an Azure VM and then setup self-hosted gateway on the VM. This way users are able to manage and maintain the underlying VM and avoid events like VM Guest OS upgrade during business hours that would have caused service disruptions on developer tier APIM.

Just a quick thought...

Finding the Right Page number in PDFs with AI Search

samsarka — Mon, 11 Aug 2025 09:10:39 GMT

Why Page Numbers Matter in AI Search

When users search for content within large PDFs—such as contracts, manuals, or reports—they often need to know not just what was found, but where it was found. Associating search results with page numbers enables:

Contextual navigation within documents.
Precise citations in knowledge bases or chatbots.
Improved user trust in AI-generated responses.

Prerequisites for Azure Blob Storage & Azure AI Search Setup Summary

1. Azure Blob Storage

A container is configured to store PDF files.

2. Appropriate permissions:

The AI search service must have Storage Blob Data Reader access to the container. If using RBAC, ensure the managed identity is properly assigned.

Ref:

Technical Approaches to Extract Page Numbers using AI search

1. Adding a Skillset: Document Cracking and index projection for parent-child indexing

The first step in skillset execution is document cracking, which separates text and image content. A common use case for Text Merger is merging the textual representation of images—such as OCR output or image captions—into the content field of a document. This is especially useful for PDFs or Word documents that combine text with embedded images. This ensures that the final enriched document includes all relevant textual data, regardless of its original format, and improves the accuracy of downstream search and analysis.

an index projection specifies how parent-child content is mapped to fields in a search index for one-to-many indexing.

{ "@odata.etag": "\"0x8DDD58F12B5D0B9\"", "name": "pagenumskillset", "description": "Skillset to feed document to OCR skill and use Index Projection to split the content page wise", "skills": [ { "@odata.type": "#Microsoft.Skills.Vision.OcrSkill", "name": "#1", "context": "/document/normalized_images/*", "lineEnding": "Space", "defaultLanguageCode": "en", "detectOrientation": true, "inputs": [ { "name": "image", "source": "/document/normalized_images/*", "inputs": [] } ], "outputs": [ { "name": "text", "targetName": "text" }, { "name": "layoutText", "targetName": "layoutText" } ] } ], "cognitiveServices": { "@odata.type": "#Microsoft.Azure.Search.DefaultCognitiveServices" }, "indexProjections": { "selectors": [ { "targetIndexName": "pagenumidx", "parentKeyFieldName": "ParentKey", "sourceContext": "/document/normalized_images/*", "mappings": [ { "name": "DocText", "source": "/document/normalized_images/*/text", "inputs": [] }, { "name": "DocName", "source": "/document/metadata_storage_name", "inputs": [] }, { "name": "DocURL", "source": "/document/metadata_storage_path", "inputs": [] }, { "name": "PageNum", "source": "/document/normalized_images/*/pageNumber", "inputs": [] } ] } ], "parameters": { "projectionMode": "skipIndexingParentDocuments" } } }

2. Defining Index definition

An index is defined by a schema and stored within the search service, ensuring millisecond response times by decoupling from external data sources. Except for indexer-driven scenarios, the search service never queries the original data directly making it ideal for high-performance search applications.

{ "@odata.etag": "\"0x8DDD58E7F7CF595\"", "name": "pagenumidx", "fields": [ { "name": "ID", "type": "Edm.String", "searchable": true, "filterable": false, "retrievable": true, "stored": true, "sortable": false, "facetable": false, "key": true, "analyzer": "keyword", "synonymMaps": [] }, { "name": "DocText", "type": "Edm.String", "searchable": true, "filterable": false, "retrievable": true, "stored": true, "sortable": false, "facetable": false, "key": false, "synonymMaps": [] }, { "name": "DocName", "type": "Edm.String", "searchable": true, "filterable": true, "retrievable": true, "stored": true, "sortable": true, "facetable": true, "key": false, "synonymMaps": [] }, { "name": "PageNum", "type": "Edm.Int32", "searchable": false, "filterable": true, "retrievable": true, "stored": true, "sortable": true, "facetable": false, "key": false, "synonymMaps": [] }, { "name": "DocURL", "type": "Edm.String", "searchable": true, "filterable": true, "retrievable": true, "stored": true, "sortable": false, "facetable": false, "key": false, "analyzer": "standard.lucene", "synonymMaps": [] }, { "name": "ParentKey", "type": "Edm.String", "searchable": true, "filterable": true, "retrievable": true, "stored": true, "sortable": false, "facetable": false, "key": false, "analyzer": "keyword", "synonymMaps": [] } ], "scoringProfiles": [], "suggesters": [], "analyzers": [], "normalizers": [], "tokenizers": [], "tokenFilters": [], "charFilters": [], "similarity": { "@odata.type": "#Microsoft.Azure.Search.BM25Similarity" }, "semantic": { "defaultConfiguration": "my_semantic_cfg", "configurations": [ { "name": "my_semantic_cfg", "flightingOptIn": false, "rankingOrder": "BoostedRerankerScore", "prioritizedFields": { "titleField": { "fieldName": "DocName" }, "prioritizedContentFields": [ { "fieldName": "DocText" } ], "prioritizedKeywordsFields": [ { "fieldName": "DocName" }, { "fieldName": "ID" } ] } } ] } }

3.Using Azure AI Search Indexer with OCR and ImageAction

Azure AI Search allows you to extract page-level data by configuring the indexer with:

This setting renders each PDF page as a separate image, which can then be processed using the OcrSkill. The OCR output can be mapped to a collection field, where each item corresponds to a page's text. This method enables you to infer page numbers based on the position of matched content in the collection.

{ "@odata.context": "https://searchinstancename.search.windows.net/$metadata#indexers/$entity", "@odata.etag": "\"0x8DDD58F3760067D\"", "name": "indexer-pagenum", "description": null, "dataSourceName": "azureblob-1754401027271-datasource", "skillsetName": "pagenumskillset", "targetIndexName": "pagenumidx", "disabled": null, "schedule": null, "parameters": { "batchSize": null, "maxFailedItems": null, "maxFailedItemsPerBatch": null, "configuration": { "dataToExtract": "contentAndMetadata", "parsingMode": "default", "imageAction": "generateNormalizedImagePerPage", "pdfTextRotationAlgorithm": "none" } }, "fieldMappings": [], "outputFieldMappings": [], "cache": null, "encryptionKey": null }

Validation and Conclusion

You can leverage Search Explorer to view the output which will look like below:

{ "@search.score": 1, "ID": "", "DocText": "the PDF content ", "DocName": "docname.pdf", "PageNum": 33, "DocURL": "https://storageaccoutname.blob.core.windows.net/containername/docname.pdf", "ParentKey": "sammple key" }

Hope this help in your requirement of getting Page Number from PDF using AI search

How to enable alerts in Batch especially when a node is encountering high disk usage

lakshmijakka — Tue, 05 Aug 2025 14:11:56 GMT

Batch users often encounter issues like nodes suddenly gets into unusable state due to high CPU or Disk usage. Alerts allow you to identify and address issues in your system. This blog will focus on how users can enable alerts when the node is consuming high amount of disk by configuring the threshold limit. With this user can get notified beforehand when the node gets into unusable state and pre-emptively takes measures to avoid service disruptions.

The task output data is written to the file system of the Batch node. When this data reaches more than 90 percent capacity of the disk size of the node SKU, the Batch service marks the node as unusable and blocks the node from running any other tasks until the Batch service does a clean up. The Batch node agent reserves 10 percent capacity of the disk space for its functionality. Before any tasks are scheduled to run, depending on the capacity of the Batch node, it's essential to keep enough space on the disk.

Best practices to follow to avoid issues with high disk usage in Azure Batch:

When the node is experiencing high disk usage, as an initial step you can RDP to the node and check how most of the space is consumed. You can check which apps and files that are consuming high disk and check if these can be deleted.

A node can experience high disk usage on OS disk or Ephemeral disk. Ephemeral disk contains all the files related to task working directory like the task output file or resource files whereas OS disk is different. The default operating system (OS) disk is usually 127 GiB only in Azure and this cannot be changed. In Batch pools using custom image, users might need to expand the OS disk when the node consumes high OS disk.

Expand virtual hard disks attached to a Windows VM in an Azure - Azure Virtual Machines | Microsoft Learn

After you have allocated extra disk on the custom image VM, you can create a new pool with the latest image.

If you want to clear manually files on node, please refer Azure Batch node gets stuck in the Unusable state because of configuration issues - Azure | Microsoft Learn

Switch to higher VM SKU

In some cases, just creating a new pool with higher VM SKU than the existing VM SKU will suffice and avoid any issues with node.

Save Task data

A task should move its output off the node it's running on, and to a durable store before it completes. Similarly, if a task fails, it should move logs required to diagnose the failure to a durable store.

It is users’ responsibility to ensure the output data is moved to a durable store before the node or job gets deleted.

Persist output data to Azure Storage with Batch service API - Azure Batch | Microsoft Learn

Clear files

If a retentionTime is set, Batch automatically cleans up the disk space used by the task when the retentionTime expires. i.e. the task directory will be retained for 7 days unless the compute node is removed or the job is deleted. This action helps ensure that your nodes don't fill up with task data and run out of disk space. Users can set this to low value to ensure output data is deleted immediately.

In some scenarios, the task gets triggered from ADF pipeline that is integrated with Batch. The retention time for the files submitted for custom activity. Default value is 30 days. Users can set the retention time in custom activity settings from ADF pipeline.

Now let’s see how to get notified when a Batch node experiences high disk usage.

Step 1: You are first required to follow below doc to integrate Azure Monitor in Batch nodes. The Azure Monitor service collects and aggregates metrics and logs from every component of the node.

Integrating Azure Monitor in Azure Batch to monitor Batch Pool nodes performance | Microsoft Community Hub

Step 2:

Once the AMA is configured, you can navigate to the VMSS in portal for which you enable metrics. Go to Metrics section and select Virtual Machine Guest from Metrics Namespace.

Step 3: From the metrics dropdown you can check metrics for the performance counter you wish.

Step 4: Now navigate to Alerts section from Menu and create alert rule.

Step 5: Here you can select any performance counter as you wish for which you want to receive alerts. Below shows creating a signal based on percentage free space that is available on VMSS.

Step 6: Once you select the signal it will ask you to provide other details for alert logic. Below snapshot shows alerts triggered when average of percentage free space available on VMSS instances is less than or equal to 20%. This alert evaluates for every one hour and check the average for the past one hour.

Step 7: You can proceed with the next steps and configure your email address and Alert rule description to receive notifications.

You can refer to below document for more information on alerts.

Create Azure Monitor metric alert rules - Azure Monitor | Microsoft Learn

In this way users can enable alerts to get notifications based on metrics for their Batch nodes. Below is a sample email alert notification.

Converting Page or Append Blobs to Block Blobs with ADF

SaikumarMandepudi — Fri, 01 Aug 2025 10:57:37 GMT

In certain scenarios, a storage account may contain a significant number of page blobs classified under the hot access tier that are infrequently accessed or retained solely for backup purposes. To optimise costs, it is desirable to transition these page blobs to the archive tier. However, as indicated in the following documentation - https://learn.microsoft.com/en-us/azure/storage/blobs/access-tiers-overview the ability to set the access tier is only available for block blobs; this functionality is not supported for append or page blobs.

The Azure blob storage connector in Azure data factory is capable of copying blobs from block, append, or page blobs and copying data to only block blobs. https://learn.microsoft.com/en-us/azure/data-factory/connector-azure-blob-storage?tabs=data-factory#supported-capabilities

Note: No extra configuration is required to set the blob type on the destination. By default, the ADF copy activity creates blobs as Block Blobs.

In this blog, we will understand how to make use of Azure Data Factory to copy the page blobs to block blobs. Please note that this is applicable to append blobs as well.

Let’s take a look at the steps ahead

Step 1: Creating ADF instance

Create an Azure data factory resource in the Azure portal referring to the following document - https://learn.microsoft.com/en-us/azure/data-factory/quickstart-create-data-factory
After creation, click on "Launch Studio" as shown below

Step 2: Creating datasets

Create two datasets by navigating to Author -> Datasets -> New dataset. These datasets are used in source and sink for the ADF copy activity

Select "Azure blob storage" -> click on continue -> select "binary" and continue

Step 3: Creating Linked service

Create a new linked service and provide the storage account name which contains page blobs

Provide the file path where the page blobs are located.
You would also need to create another dataset for destination. Repeat the steps from 3 to 6 to create another destination dataset to copy the blobs to the storage account as block blobs.

Note: You can use same or different storage account for the destination dataset. Set it as per your requirements.

Step 4: Configuring a Copy data pipeline

Once the two datasets are created, now create a new pipeline and under "Move and Transform" section, drag and drop the "Copy data" activity as shown below.

Under the Source and Sink sections from the drop down, select the source and destination datasets respectively which were created in the previous steps. Select the “Recursively” option and publish the changes.
- Source:

- Sink:

Note: You can configure the filters and copy behaviour as per your requirements.

Step 5: Debugging and validating

Now as the configuration is completed, click on "Debug".
If the pipeline activity ran successfully, you should be able to see "succeeded" status in the output section as below.

Verify the blob type of the blobs in the destination storage account and it should show as block blob and access tier as Hot.

After converting the blobs to block blobs, several methods are available to change their access tier to archive. These include implementing a blob lifecycle management policy, utilizing storage actions, or by using Az CLI or PowerShell scripts.

Conclusion

Utilising ADF enables the conversion of page or append blobs to block blobs, after which any standard method such as LCM policy or storage actions may be used to change the access tier to archive. This strategy offers a more streamlined and efficient solution compared to developing custom code or scripts.

Reference links:

Rehydrating Archived Blobs via Storage Task Actions

ankitsah — Wed, 09 Jul 2025 06:12:02 GMT

Azure Storage Actions is a fully managed platform designed to automate data management tasks for Azure Blob Storage and Azure Data Lake Storage. You can use it to perform common data operations on millions of objects across multiple storage accounts without provisioning extra compute capacity and without requiring you to write code.

Storage task actions can be used to rehydrate the archived blobs in any tier as required. Please note there is no option to set the rehydration priority and is defaulted to Standard one as of now.

Note :-

Azure Storage Actions are generally available in the following public regions: https://learn.microsoft.com/en-us/azure/storage-actions/overview#supported-regions

Azure Storage Actions is currently in PREVIEW in following reions. Please refer: https://learn.microsoft.com/en-us/azure/storage-actions/overview#regions-supported-at-the-preview-level

Below are the steps to achieve the rehydration :-

Create a Task :-
1. In the Azure portal, search for Storage tasks. Under Services, select Storage tasks - Azure Storage Actions.
2. On the Azure Storage Actions | Storage Tasks page, select Create
3. Fill in all the required details and click on Next to open the Conditions page.
4. Add the conditions as below if you want to rehydrate to Cool tier.
Add the Assignment :-
1. Select Add assignment.
2. In the Select scope section, select your subscription and storage account and name the assignment.
3. In the Role assignment section, in the Role drop-down list, select the Storage Blob Data Owner to assign that role to the system-assigned managed identity of the storage task.
4. In the Filter objects section, specify the filter if you want this to run on some specific objects or the whole storage account.
5. In the Trigger details section, choose the runs of the task and then select the container where you'd like to store the execution reports.
6. Select Add.
7. In the Tags tab, select Next.
8. In the Review + Create tab, select Review + create.
9. When the task is deployed, Your deployment is complete page appears.
10. Select Go to resource to open the Overview page of the storage task.
Enable the Task Assignment :-

Storage task assignments are disabled by default. Enable assignments from the Assignments page.
1. Select Assignments, select the assignment, and then select Enable.
  
  The task assignment is queued to run and will run at the specified time.
Monitoring the runs :-
After the task completes running, you can view the results of the run.
1. With the Assignments page still open, select View task runs.
2. Select the View report link to download a report.
  You can also view these comma-separated reports in the container that you specified when you configured the assignment.

Reference Links :-

Integrating Azure Monitor in Azure Batch to monitor Batch Pool nodes performance

lakshmijakka — Thu, 03 Jul 2025 08:25:59 GMT

In Azure Batch, to monitor the node performance like CPU or Disk usage users are required to use Azure monitor. The Azure Monitor service collects and aggregates metrics and logs from every component of the node. Azure Monitor provides you with a view of availability, performance, and resilience. When you create an Azure Batch pool, you can install any of the following monitoring-related extensions on the compute nodes to collect and analyse data.

Previously users have leveraged Batch Insights to get system statistics for Azure Batch account nodes, but it is deprecated now and no longer supported.

The Log Analytics agent virtual machine (VM) extension installs the Log Analytics agent on Azure VMs and enrols VMs into an existing Log Analytics workspace. The Log Analytics agent is on a deprecation path and won't be supported after August 31, 2024. Migrate to Azure Monitor Agent from Log Analytics agent - Azure Monitor | Microsoft Learn

Azure Monitor Agent (AMA) now replaces the Log Analytics agent.

Install and Manage the Azure Monitor Agent - Azure Monitor | Microsoft Learn

Important!

Currently Azure Monitor in Batch Pool is only supported for Batch accounts that are created with Pool allocation Mode in User subscription mode only. Batch accounts that are created with Pool allocation Mode in Batch Service are not supported. As in Batch Service mode, nodes will be created in Azure subscriptions and users do not have access to these subscriptions, enabling data collection for these nodes is not possible.

This article will focus on how you can use to install and configure the Azure Monitor Agent extension on Azure Batch pool nodes.

Note : Extensions cannot be added to an existing pool. Pools must be recreated to add, remove, or update extensions. Currently the Batch pool with user assigned Managed Identity and extension is only supported by ARM template and REST API call. Creating a pool with extension is unsupported in Azure Portal. Creating a pool with user assigned Managed Identity is unsupported in Az PowerShell module and Azure CLI.

To use the templates below, you'll need to follow below prerequisites:

To create a user-assigned managed identity. A managed identity is required for Azure Monitor agent to collect and publish data.
To configure data collection for Azure Monitor Agent, you must also configure or deploy Resource Manager template data collection rules and associations.

Step 1: Create a pool with AMA extension

Below is sample JSON template to create a pool with AMA extension enabled for Windows server.

{
"name": "poolextmon",
"type": "Microsoft.Batch/batchAccounts/pools",
"properties": {
"allocationState": "Steady",
"vmSize": "STANDARD_D2S_V3",
"interNodeCommunication": "Disabled",
"taskSlotsPerNode": 1,
"taskSchedulingPolicy": {
"nodeFillType": "Pack"
},
"deploymentConfiguration": {
"virtualMachineConfiguration": {
"imageReference": {
"publisher": "microsoftwindowsserver",
"offer": "windowsserver",
"sku": "2019-datacenter",
"version": "latest"
},
"nodeAgentSkuId": "batch.node.windows amd64",
"extensions": [
{
"name": "AzureMonitorAgent",
"publisher": "Microsoft.Azure.Monitor",
"type": "AzureMonitorWindowsAgent",
"typeHandlerVersion": "1.0",
"autoUpgradeMinorVersion": true,
"enableAutomaticUpgrade": true,
"settings": {
"authentication": {
"managedIdentity": {
"identifier-name": "mi_res_id",
"identifier-value": "/subscriptions/xxxxx/resourceGroups/r-xxxx/providers/Microsoft.ManagedIdentity/userAssignedIdentities/usmi"
}
}
}
}
]
}
},
"scaleSettings": {
"fixedScale": {
"targetDedicatedNodes": 1,
"targetLowPriorityNodes": 0,
"resizeTimeout": "PT15M"
}
},
"currentDedicatedNodes": 1,
"currentLowPriorityNodes": 0,
"targetNodeCommunicationMode": "Default",
"currentNodeCommunicationMode": "Simplified"
}
}

Below is sample JSON template to create a pool with AMA extension enabled for Linux server.

{
"name": "poolextmon",
"type": "Microsoft.Batch/batchAccounts/pools",
"properties": {
"allocationState": "Steady",
"vmSize": "STANDARD_D2S_V3",
"interNodeCommunication": "Disabled",
"taskSlotsPerNode": 1,
"taskSchedulingPolicy": {
"nodeFillType": "Pack"
},
"deploymentConfiguration": {
"virtualMachineConfiguration": {
"imageReference": {
"publisher": "canonical",
"offer": "0001-com-ubuntu-server-jammy",
"sku": "22_04-lts",
"version": "latest"
},
"nodeAgentSkuId": "batch.node.ubuntu 22.04",
"extensions": [
{
"name": "AzureMonitorAgent",
"publisher": "Microsoft.Azure.Monitor",
"type": "AzureMonitorLinuxAgent",
"typeHandlerVersion": "1.0",
"autoUpgradeMinorVersion": true,
"enableAutomaticUpgrade": true,
"settings": {
"authentication": {
"managedIdentity": {
"identifier-name": "mi_res_id",
"identifier-value": "/subscriptions/xxxxxx/resourceGroups/r-xxx/providers/Microsoft.ManagedIdentity/userAssignedIdentities/usmi"
}
}
}
}
]
}
},
"scaleSettings": {
"fixedScale": {
"targetDedicatedNodes": 1,
"targetLowPriorityNodes": 0,
"resizeTimeout": "PT15M"
}
},
"currentDedicatedNodes": 1,
"currentLowPriorityNodes": 0,
"targetNodeCommunicationMode": "Default",
"currentNodeCommunicationMode": "Simplified"
}
}

Once the pool is created you can verify if the extension is installed on pool from portal.

Step 2: Create Log analytics workspace

You are required to have a log analytics workspace where the data will be sent.

Step 3: Create a data collection rule (DCR)

Create a DCR to collect data by using the Azure portal by following below document. You can refer to below document on how to create a DCR. Below document also talks about the types of data you can collect from a VM client with Azure Monitor and where you can send that data.

Collect data from virtual machine client with Azure Monitor - Azure Monitor | Microsoft Learn

Once the VM is associated to DCR, you can check the computers connected in Log Analytics workspace.

To verify that the agent is operational and communicating properly with Azure Monitor check the Heartbeat for the VM.

To verify that data is being collected in the Log Analytics workspace, check for records in the Perf table.

To verify that data is being collected in Azure Monitor Metrics, select Metrics from the virtual machine in the Azure portal. Navigate to the VMSS from portal and select Virtual Machine Guest (Windows) or azure.vm.linux.guestmetrics for the namespace and then select a metric to add to the view.

Lifecycle Management of Blobs (Deletion) using Automation Tasks

thakurmishra — Thu, 22 May 2025 04:17:03 GMT

Background:

We often encounter scenarios where we need to delete blobs that have been idle in a storage account for an extended period. For a small number of blobs, deletion can be handled easily using the Azure Portal, Storage Explorer, or inline scripts such as PowerShell or Azure CLI.

However, in most cases, we deal with a large volume of blobs, making manual deletion impractical. In such situations, it's essential to leverage automation tools to streamline the deletion process. One effective option is using Automation Tasks, which can help schedule and manage blob deletions efficiently.

Note: Behind the scenes, an automation task is actually a logic app resource that runs a workflow. So, the Consumption pricing model of logic-app applies to automation tasks.

Scenario’s where “Automation Tasks” are helpful:

You have a requirement to automate deletion of blobs which are older than a specific time, in days, weeks or months.
You don’t want to put in much manual effort rather have simple UI based steps to achieve your goal
You have System containers, and you want to action on it.
We have “LCM (Life Cycle management)” which too can be leveraged by users to automation deletion of older blobs; however LCM cannot be used to delete blobs from System containers.
You have to work on page blobs.

Setup “Automation Tasks”:

Let’s walk through on how to achieve our goal.

Navigate to the desired storage account and scroll down to the “Automation” section and select the “Tasks” blade and then click on “Add Task” from the top panel or bottom panel (highlighted in image).

On the next page click the “Select” (highlighted image)
The new page which opens up should look as below, however there isn’t anything we are doing. So let’s just click on the “Next : Configure” (highlighted in image) and move to the next screen.
The new page opens needs to be filled as per your requirement. I have added a sample. You can also use it on your own containers as well.
- 'sample' is a folder inside container '$web'

The “Expiration Age” field means that Blobs older than these number of days needs be deleted. In above screenshot, blobs older than 180 days would be deleted.

Similarly, we can configure values in weeks or months as well.

Once we are through with the steps proceed with creation of the task.

Once task is created it looks as below:
You can click on the “View Run” to see run history.
In-case you want to modify the task, click on your tasks name. For example in above screenshot I can modify by clicking “mytask” link and re-configure the task.

Now this isn’t sufficient. We will update some of the steps which were used to create the Logic-app. Hence we would need to edit some steps and save those before re-running the app.

a) Go the logic app and navigate to the “Logic App Designer” blade

b) Now click on the “+” sign as shown below and “Add an Action”

c) Once the new page opens up, search for “List Blobs (v2)” and select it

d) Choose the “Enter custom value” and enter your storage account name

e) The values would like as shown below

f) Now let's navigate to the “For Each” condition

g) We need to delete the “Delete blob” too and replace with “Delete blob (V2)”

h) The “Delete Blob (V2)” looks like as below

i) With all steps ready, lets save the logic app and click on “Run”. You should observe the run passing successfully.

Impact due to Firewall:

The above steps for works when your storage account is configured for public access.

However, when firewall is enabled, you would need to provide the necessary permissions, else you are going to encounter 403 "Authorization Failure" errors. There would be no issue to create the task, but you will see failures when you check the runs. Example:

To overcome this limitation, you need to navigate to your logic app and generate a managed identity for the app and provide the identity “Storage Blob Data Contributor” role.

Step1. Enable Managed Identity:

In Azure Portal, go to your Logic App resource.
Under Settings, select Identity.
In the Identity pane, under System assigned, select On and Save.
This step registers the system-assigned identity with Microsoft Entra ID, represented by an object ID.

Step2. Assign Necessary Role:

Open the Azure Storage Account in Azure Portal.
Select Access control (IAM) > Add > Add role assignment.
Assign a role like 'Storage Blob Data Contributor', which includes write access for blobs in an Azure Storage container, to the managed identity.
Under Assign access to, select Managed identity > Add members, and choose your Logic App's identity
Save and refresh and you see the new role configured to your storage account

Remember that, if the Storage Account and logic app are in different region you should add another step in the firewall of storage account. You need to whitelist the logic app instance in “Resource instances” list as shown below:

Conclusion:

The multiple ways to action on blobs are provided for your convenience. Depending on your requirement, feasibility & other factors like comfortability with the feature or pricing too would certainly influence your decisions.

However, in-case you want to action upon System containers like $logs or $web, “Automation Tasks” are one of the most helpful feature which you can use and achieve your goal.

Note: At the time of writing this blog this feature is still in preview. So ensure to check if there are any limitations which might impact you before implementing it in your Production environment.

References:

Create automation tasks to manage and monitor Azure resources - Azure Logic Apps | Microsoft Learn

Optimize costs by automatically managing the data lifecycle - Azure Blob Storage | Microsoft Learn

[AI Search] LockedSPLResourceFound error when deleting AI Search

SungGun_Lee — Wed, 21 May 2025 05:59:13 GMT

Are you unable to delete AI Search with the following error?

LockedSPLResourceFound :Unable to verify management locks on Resource '$Resource_Path '. If you still want to delete the search service, manually delete the SPL resource first and try again.

If you are, this is the right place for you to find a quick resolution! Keep on reading through.

[Solution – Delete the Shared Private Link]

The error message will appear if you still have Shared Private Link configured in AI Search. AI Search will not let you delete the resource unless you delete the Shared Private Link first.

You must delete the Shared Private Link in the Portal manually.
Move to Settings > Networking > Shared private access tab.

Once the Shared Private Links are all deleted, please try again to delete the AI Search.

Also please give at least 15 minutes for the Shared Private Links to be deleted completely as it may take longer.

[Extra - I tried to delete Shared Private Links but it’s been pending for a long while]

There are occasions where you will see your Shared Private Links are in a state of being deleted for a long time as below. (For example 3 hours plus or more)

In this case, please open a case to our Support team mentioning you are having an issue with deleting AI Search due to Shared Private Link not being deleted properly. Our team will take care of the issue from that point on!

Introducing Azure SRE Agent

dchelupati — Tue, 18 Nov 2025 06:09:11 GMT

Today we’re thrilled to introduce Azure SRE Agent, an AI-powered tool that makes it easier to sustain production cloud environments. SRE Agent helps respond to incidents quickly and effectively, alleviating the toil of managing production environments. Overall, it results in better service uptime and reduced operational costs. SRE Agent leverages the reasoning capabilities of LLMs to identify the logs and metrics necessary for rapid root cause analysis and issue mitigation. Its advanced AI capabilities transform incident and infrastructure management in Azure, freeing engineers to focus on more meaningful work.

As more companies move their services online, site reliability engineering (SRE) has become crucial to keeping critical systems reliable, scalable, and cost-efficient. But SRE isn't just about fixing problems—it’s about bridging the gap between business goals and developer needs. With growing infrastructure complexity, it’s harder than ever to keep everything running smoothly while anticipating future scalability and reliability needs.

We’ve heard from SREs that they experience significant toil from repetitive live-site incident handling and log analysis tasks, with adhoc administrative tasks disrupting their workflow. Responding to incidents is stressful, as seconds matter and there’s little room for error.

SRE Agent brings the years of experience Microsoft teams gathered running the Azure cloud to your team.

SRE Agent is a new Azure service that gives site reliability engineers (SREs) and developers the tools they need to increase the speed and efficiency of incident responses, diagnostics and collaboration to resolve problems rapidly. It is seamlessly integrated with other observability and incident management tools, as well as the new coding agent in GitHub Copilot. It runs in the background 24x7, learning and monitoring the health and performance of Azure resources, handling production alerts, and partnering in incident investigations and root cause analysis (RCA) to mitigate issues faster.

Key Capabilities

SRE Agent can help make your infrastructure more secure, resilient and scalable and can help detect and respond to incidents more quickly.

Evaluating usage and performance trends

SRE Agent continuously learns about your Azure resources to build relevant context about them without the hassle of multiple tools. You can ask questions to learn about their properties, configuration, and recent changes. And you can learn about their health and performance by visualizing relevant metrics. This allows developers to quickly identify anomalies or trends that need attention.

Example Prompts:

What changed to my app in last day?
When was the last slot swap performed on my app?
What alerts should I set up for my web app?
Can you give me the overall AKS cluster usage?
What are best practices that I should setup for my app?
Visualize requests and 500 errors for last week for my app

Proactive detection and remediation of security vulnerabilities

SRE Agent continuously audits Azure resources to ensure compliance with security best practices. Currently, it checks for the use of supported TLS versions and verifies if resources have Managed Identity enabled. SRE Agent not only identifies the potential vulnerabilities but can also perform the necessary operations to update resources with your approval, bringing them into compliance.

Automated incident response and Faster root cause analysis

SRE Agent can respond to Azure Monitor alerts out of the box. You can also integrate with Incident Management tools such as PagerDuty to extend its alert-handling capabilities. With this integration, SRE Agent can:

Start investigation upon alert detection.
Access metrics, activity logs, dependencies, and dashboards to form hypotheses and determine the root cause.

Traditional RCA methods can take hours, while the SRE Agent completes them in minutes, minimizing impact and driving faster resolutions.

Incident mitigation

To mitigate incidents and bring an application back to operational state, SRE Agent can perform operations on behalf of and with approval of the user. These operations can include scaling up resources, restarting applications, and performing rollbacks to a previously working version of the app.

Close the loop with developers

Once an investigation is complete, SRE Agent creates a GitHub issue including all the details from the investigation helping developers to fix source code and prevent subsequent recurrences of an incident.

Share your thoughts

We would love to hear your feedback. Signup for preview access to SRE Agent is here. Use the “Thumbs Up” or “Thumbs Down” buttons to share your thoughts on Agent’s responses. You can also share feedback on the product via the “give us feedback” button on the top right corner inside the SRE Agent. Your input is invaluable to us as we strive to improve and support you on your Azure journey.

Additional resources

signup for preview access
Build session
Agentic DevOps Blog post
Pricing announcement (billing starts September 1 2025)

Custom Tracing in API Management

shailesh14 — Tue, 06 May 2025 18:51:06 GMT

Scenario:

In case of encountering error in API management, request tracing is an invaluable feature that serves as a debugger. It allows for tracking the flow of requests as they pass through various policy logic, providing detailed insights into the complete API Management (APIM) processing. Here is a link if you would like to read more on how to enable request tracing in API management.

However, it is the most common way to debug your API, let's assume a real-life scenario where you encounter a sporadic error or unexpected response while processing the live APIM calls and need to drill down the issue. In such cases, attaching a debugger or running request traces can be challenging, especially when the issue is intermittent or requires checking the specific code logic. This often necessitates a trial-and-error approach to reproduce the scenario and obtain the traces. To address these challenges, the APIM Trace policy can be utilized. This policy enables the addition of custom traces to Application Insights and/or to resource logs.

This blog can help users to use the APIM Trace policy which can be leveraged to sort out such kind of issues. The following snippet demonstrates how to use traces with Application Insights.

Pre-Requirements:

Before we start, we could make sure that we have existing Azure API Management service and App Insight setup for logging these requests.

Steps:

Once you are done with your APIM setup make sure your diagnostics setting is correctly setup and it is logging your API calls, you can verify the same by going to App Insight instance and running the queries for "requests" table.

Now let's consider a scenario where we may need to check on the value of remaining calls which will be set based on certain input parameters (or any other logic on your policy code). To do the same go to your API's>Design> "Your API".
I did setup the verbosity as Verbose here to log the APIM request to App insight and we will be using the trace policy code to add custom trace information to the request tracing output as shown below.
As an example, we will be capturing the traces for the selected API only. The below example is for illustration purpose only here we are trying to log remaining calls under trace variable "callremain". We will be leveraging rate limit policy to check number of calls left and adding a custom trace information. Here is a policy code for the same

<choose> <when condition="@(((string)context.Request.Headers.GetValueOrDefault("x-env","none")) =="test")"> <rate-limit-by-key calls="5" renewal-period="30" counter-key="@(context.Request.IpAddress)" increment-condition="@(context.Response.StatusCode == 200)" remaining-calls-variable-name="remainingCallsPerIP" /> <trace source="remainingCallsPerIP" severity="verbose"> <message>@(String.Format("{0} | {1}", "ip address", context.Request.IpAddress))</message> <metadata name="callremain" value="@{ var calls = context.Variables.GetValueOrDefault<Int32>("remainingCallsPerIP"); return calls.ToString(); }" /> </trace> </when> <otherwise> <rate-limit calls="10" renewal-period="20" /> </otherwise> </choose>
Once you save the above policy code run the API calls using header x-env the rate limit policy only allows 5 calls per 30 seconds here and will log the traces with message and call remain variable.
Next Go to App insight(your app insight instance) >Monitoring>Logs>traces
You can filter traces based on your Kusto like API name or custom dimension however here for the simplicity for our sample call filter is based on message contains "ip address". Please note it is the message metadata we added in our trace policy and call remain variable which you can track based on your business logic.
traces

| where message contains "ip address"
Keep calling the API, for example using VS Code or any other tool to make continuous calls to API and capture the variable "callremain". Here the trace policy helps to identify how many calls left in the rate limit policy by logging custom messages and metadata to the trace output.
This will provide a way to look deep into your code logic or if you would like to track any variables in your APIM code. You can add more metadata in the <Trace> policy as per your debug requirements. Here is another quick example to save your request and change the severity as informational in the trace log.

Conclusion: The above steps outlined will give you an idea on how to use this policy to add custom messages and metadata to the trace output, which can help in debugging and troubleshooting API's by providing detailed information about request processing steps. This includes inbound requests, backend interactions, and outbound responses. I hope this blog provided you with information to better grasp what happens when an API is called, and how trace policy can help you in troubleshooting and debugging your APIM calls.

How Networking setting of Batch Account impacts simplified communication mode Batch pool

JerryZhangMS — Fri, 02 May 2025 04:12:44 GMT

As described in our official document, the classic communication mode of Batch node will be retired on 31 March 2026. Instead, it’s recommended to use simplified communication mode while creating Batch pool.

But while user changes their Batch pool communication mode from classic to simplified and applies the necessary changes of network security group per documentation, they will find out that the node is still stuck in unusable status.

A very possible cause of this issue is due to the bad networking setting of Batch Account.

This blog will mainly talk about why networking setting can cause node using simplified communication mode stuck in unusable status and how to configure correct networking setting under different user scenarios.

Cause:

As described in this document, the difference between classic and simplified communication mode is very clear:

Classic: the Batch service initiates communication with the compute nodes.
Simplified: the compute nodes initiate communication with the Batch service.

The purpose of communication is simple: Batch service needs to receive traffic from Batch nodes to know whether a node is healthy and which status it’s in.

The difference is where the traffic initiates. If it’s initiated from Batch service side like classic communication mode, then it’s considered as outgoing traffic of your Batch Account. If it’s initiated from Batch nodes like simplified communication mode, then it’s considered as incoming traffic of your Batch Account.

The networking settings of Batch Account will only check the incoming traffic, not outgoing one. Hence if the networking setting completely disables public network access, the classic communication mode nodes will still be able to communicate with Batch service, but the simplified communication mode nodes will be unable to communicate with Batch service, which will further cause Batch service to mark this node as unusable status.

Solution:

The only important point of solution is to make sure the traffic from the simplified communication mode node is allowed by Batch Account networking setting.

Here is the diagram for different user scenarios:

*1: The resource group where the public IP address is created will be different depending on the Batch Account pool allocation mode. If it’s Batch Service, the public IP address will be created in same resource group as Virtual Network resource. If it’s User Subscription, it will be in a resource group with name AzureBatch-{GUID}-C.

*2: This scenario is as document

TIPS: In some scenarios, there will be more than 1 public IP address which can be used by a Batch pool, such as the scenario of pool with Virtual Network and own public IP address will require one additional public IP address as buffer, or the scenario of pool with Virtual Network with more than 100 nodes. In those scenarios, please remember to put all public IP addresses into allow list.

AI Resilience: Strategies to Keep Your Intelligent App Running at Peak Performance

fabiopadua — Thu, 24 Apr 2025 16:52:18 GMT

Stay Online

Reliability. It's one of the 5 pillars of Azure Well-Architect Framework.

When starting to implement and go-to-market any new product witch has any integration with Open AI Service you can face spikes of usage in your workload and, even having everything scaling correctly in your side, if you have an Azure Open AI Services deployed using PTU you can reach the PTU threshold and them start to experience some 429 response code.

You also will receive some important information about the when you can retry the request in the header of the response and with this information you can implement in your business logic a solution. Here in this article I will show how to use the API Management Service policy to handle this and also explore the native cache to save some tokens!

Architecture Reference

The Azure Function in the left of the diagram just represent and App request and can be any kind of resource (even in an On-Premisse environment). Our goal in this article is to show one in n possibilities to handle the 429 responses. We are going to use API Management Policy to automatically redirect the backend to another Open AI Services instance in other region in the Standard mode, witch means that the charge is going to be only what you use.

First we need to create an API in our API Management to forward the requests to your main Open AI Services (region 1 in the diagram).

Now we are going to create this policy in the API call request:

The first part of our job is done! Now we have an automatically redirect to our OpenAI Services deployed at region 2 when our PTU threshold is reached.

Cost consideration

So now you can ask me: and about my cost increment for using API Management?

Even if you don't want to use any other feature on API Management you can leverage of the API Management native cache and, once again using policy and AI, put some questions/answers in the built-in Redis* cache using semantic cache for Open AI services.

Let's change our policy to consider this:

<policies> <inbound> <base /> <azure-openai-semantic-cache-lookup score-threshold="0.05" embeddings-backend-id ="azure-openai-backend" embeddings-backend-auth ="system-assigned" > <vary-by>@(context.Subscription.Id)</vary-by> </azure-openai-semantic-cache-lookup> <set-backend-service base-url="<your_open_ai_region1_endpoint>" /> </inbound> <backend> <base /> </backend> <outbound> <base /> <azure-openai-semantic-cache-store duration="60" /> </outbound> <on-error> <retry condition="@(context.Response.StatusCode == 429)" count="1" interval="5" /> <set-backend-service base-url="<your_open_ai_region2_endpoint>" /> </on-error> </policies>

Now, API Management will handle the tokens inputted and use semantic equivalence and decide if its fit with cached information or redirect the request to your OpenAI endpoint. And, sometime, this can help you to avoid reach the PTU threshold as well!

* Check the tier / cache capabilities to validate your business solution needs with the API Management cache feature: Compare API Management features across tiers and cache size across tiers.

Conclusion

API Management offers key capabilities for AI that we are exploring in this article and also others that you can leverage for your intelligent applications. Check it out on this awesome AI Gateway HUB repository

At least but not less important, dive in API Management features with experts in the field inside the API Management HUB.

Thanks for reading and Happy Coding!

Streaming and Analyzing Azure Storage Diagnostic Logs via Event Hub using Service Bus Explorer

jainsourabh — Thu, 24 Apr 2025 11:27:12 GMT

Monitoring Azure Storage operations is crucial for ensuring performance, compliance, and security. Azure provides various options to collect and route diagnostic logs. One powerful option is sending logs to Azure Event Hub, which allows real-time streaming and integration with external tools and analytics platforms.

In this blog, we’ll walk through setting up diagnostic logging for an Azure Storage account with Event Hub as the destination, and then demonstrate how to analyse incoming logs using Service Bus Explorer.

Prerequisites

Before we begin, make sure you have the following set up:

1. Azure Event Hub Configuration

An Event Hub namespace and instance set up in your Azure subscription.

2. Service Bus Explorer Tool

We'll use Service Bus Explorer to connect to Event Hub and analyse log data.

Download and Setup:

Go to the official GitHub page:
https://github.com/paolosalvatori/ServiceBusExplorer
Download the latest release .zip file from the Releases section.
Extract the contents and launch ServiceBusExplorer.exe (no installation needed).

Note:- This is a Windows-only tool. Make sure .NET runtime is installed on your system.

3. Event Hub Connection String

You’ll need a connection string with appropriate permissions to connect via Service Bus Explorer:

Azure Portal → Navigate to your Event Hub Namespace → Shared Access Policies
Select RootManageSharedAccessKey with Managed rights
Copy the Connection string–primary key

Note:- Ensure the connection string includes the Entity Path if you're targeting a specific Event Hub.

4. Ensure Diagnostic Logging is Enabled for Azure Storage

To stream logs into Event Hub, make sure that diagnostic logging is configured properly on your Azure Storage account.

Steps:

Navigate to your Storage Account in the Azure Portal.
Go to Monitoring > Diagnostic settings.
Click Add diagnostic setting or edit an existing one.
Select the required log categories: Blob, Table, Queue, File (as needed).
Set Event Hub as the destination.

Let's take a look at the steps below for configuring Service Bus Explorer and Reviewing the logs ahead

Step 1: Connect to Event Hub Using Service Bus Explorer

To analyze the streamed logs, we will use Service Bus Explorer, a powerful tool for inspecting messages within Azure Event Hub.

Open Service Bus Explorer.

Click on File > Connect.

Step 2: Provide Event Hub Connection String

Paste your Event Hub-compatible connection string at the Namespace level from the Portal which has Manage permissions and paste it in the right-hand text field under Connection Settings and click on Save.

Step 3: View Available Event Hubs

Once connected, you'll see a list of Event Hubs under your namespace.
Expand the Event Hub you configured for diagnostic logs.

Step 4: Start Listening to the Consumer Group

Right-click on your Consumer Group (usually $Default).
Select "Create Consumer Group Listener" to begin listening for incoming data.

Step 5: Enable Verbose Logging

Check the Verbose option to view more detailed information about the incoming messages, including metadata.

Step 6: View and Analyse Events

Navigate to the Events tab at the top.
Click on Start to begin streaming live log events.

You’ll now see:

Metadata: Sequence Number, Offset, Enqueued Time (UTC), Partition Info.
Payload: The actual log data coming from Azure Storage (in JSON format).

🧠 Tip: You can copy and analyse this payload further using tools like Power BI, Stream Analytics, or even a custom parser.

Conclusion

Streaming diagnostic logs to Event Hub gives you flexibility in how you handle monitoring data. Whether you're integrating with external SIEM solutions, triggering alerts, or performing deep analytics, Event Hub provides a real-time backbone for diagnostics.

Using Service Bus Explorer, you can quickly validate and analyse the incoming logs to ensure your setup is working and to inspect what's being captured.

Reference Link :-

https://learn.microsoft.com/en-us/azure/azure-monitor/essentials/diagnostic-settings

Tips for Migrating Azure Event Hub from Standard to Basic Tier Using Scripts

sudeshna — Thu, 24 Apr 2025 08:34:15 GMT

Introduction

What are Event Hubs?

Azure Event Hub is a big data streaming platform and event ingestion service by Microsoft Azure. It’s designed to ingest, buffer, store, and process millions of events per second in real time.

Feature Comparison

The Standard tier of Azure Event Hubs provides features beyond what is available in the Basic tier. The following features are included with Standard:

Feature	Basic Tier	Standard Tier
Capture Feature	❌ Not available	✅ Available
Virtual Network Integration	❌ Not available	✅ Available
Auto-Inflate	❌ Not available	✅ Available
Consumer Groups	Limited (only 1 group)	Up to 20 consumer groups
Message Retention	Up to 1 day	Up to 7 days

Many organizations or users choose to downgrade their Event Hubs from the Standard to the Basic tier to reduce costs or for other reasons.

However, it's important not to overlook that certain features available in the Standard tier are not supported in the Basic tier. These differences mentioned above should be carefully reviewed before making the switch.

So, while you try to switch from the Portal, you will be able to see that Basic tier is enabled, and you are good to migrate from Standard to Basic

However, when you start migrating from the Portal you will see the error below which means your Event Hub capture feature is enabled and you need to disable it first before migrating.

Failed updating Event Hubs namespace with error: Code: 'MessagingGatewayConflict' with Message: 'Namespace tier cannot be downgraded, as Archive is not available in Basic tier. TrackingId:xxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxx-xxxxxx-xxxxxxxxxx-xxxxxxxexx_xxx, SystemTracker:eventhubnamespacename.servicebus.windows.net:$tenants/eventhubnamespacename.

When running a script for multiple namespaces, this error may not appear. You might incorrectly assume the migration succeeded, while the status will remain 'Not Active.' This can silently break functionality.

Current Limitation

If Capture Feature is enabled, it should not allow to migrate on the first place.
You must have created multiple Consumer groups for Standard tier. However, after migration, you can use only 1 Consumer Group that is $default for the basic tier.
If Auto-inflate is enabled, it will ask you to disable first and then start the migration.

In Ideal scenario, you should not be able to pull events from other consumer groups as Basic only offers 1 consumer group per entity for Basic.

However, if you try to pull events from other consumer groups it does not throw any throw Runtime Error.

This bug has been identified by our Internal Product Group team, and they are working on it to fix this bug and release it sooner.

Before implementing this workaround addressed below described in this article, I recommend checking the status of this feature request. Event Hubs is continuously evolving, and by the time you're reading this, the limitation might have been addressed in the upcoming weeks.

Workaround

You need to disable Capture feature first and check the Status = Active. Then start the migration process and check again the Status = Active.
Delete all the other Consumer Groups which you created in Standard tier to avoid confusion.
Do not use any other Consumer groups other than $default in your receiver applications while pulling the Events from the Event Hub.

Conclusion

"Thanks for reading! I hope this article helps clarify the nuances of migrating from the Standard to Basic tier in Azure Event Hubs. As the platform evolves, keep an eye on official documentation and Azure updates. This limitation may be resolved soon, making migration more seamless in the future."

Automate creation of work items in ADO and Export/Import workflow packages

Gayatri_Ram — Wed, 16 Apr 2025 20:25:09 GMT

This article would create multiple work items (tasks) for any specific User Story in a particular Backlog with a certain TAG value in Azure Devops. This article would also show steps to Export/Import a workflow package.

PART 1: Create multiple items for a parent User Story
This article uses Power Automate to do the same.
There are certain Pre-requisites that have to be fulfilled,
1. Be a part of an Azure Devops organization, have a project created along with some User Stories.
2. Have access to Power Automate to create workflows, and make sure you are able to add connections from Automate to ADO (Not to worry we would check that in the below steps)

Now go ahead and
Follow the below steps to achieve the purpose.

Step1: Open power automate portal and Click on "My Flows"
Link: https://make.powerautomate.com

Step2: Click on "New flow" and on "Automated cloud flow"

Step3: Choose an appropriate name and search for ADO trigger "When a work item is assigned"

Step4: Now with the Create option, it would depict invalid parameters like below

Next step would resolve this.

Step5: Click on the workflow item and following tab would appear, fill in the organization and other details

Now click on "Show All" options and fill in the Area Path, Iteration Path, Created By and Type

Step6: Now once the previous step is complete, click on the Plus symbol and select 'Add a condition' option

This condition is the part where we add the specific tag value and make sure that User stories with only these tag values are picked up and not all the others

Step7: Choose the left entry as 'Tags'

and enter value here in this case , 'Custom' is used.

Intermediately keeping saving the workflow to avoid any losses.

Step8: Now lets add the work item details, Under the True condition (shown as below), click on plus symbol and 'Add an action'

Click on 'Azure Devops' - 'Create a work item'

Step 9: Click on the item, and fill in ADO details

Important part is to click on ShowAll

and make sure the option Link URL is chosen correctly.

Step 10: Choose 'Hierarchy-Reverse' in the Link Type

Step 11: Now lets create parallel tasks for the parent User Story
Click 'Add a parallel branch', similar to the previous steps
Click on 'Azure Devops' - 'Create a work item'

Step 12: Save and Test the workflow

Now the workflow would create multiple 'tasks' for a User Story with a Tag value 'Custom'

Lets test the workflow

The run was successful, and the User story has child 'tasks'

All the tasks, have parent Area Path and Iteration Path

PART 2: In the second section of the document, lets learn about how to export a workflow and import the same.

Step1: Click on Export and choose 'package(.zip)'

Step2: Enter details and click on 'Export'

Now the package would be downloaded to local

Importing exported package

Step3: Click on 'my flows' and Import option

Step4: The next would be as below

And once Import is clicked the workflow would get created from the exported package.

To conclude , we have tested the workflow for User Stories with a specific 'Tag' value . The workflow creates multiple tasks as child items. We also tested the export/import of workflow packages.

Lease Management in Azure Storage & Common troubleshooting scenarios

rpadi450 — Wed, 09 Apr 2025 12:01:45 GMT

The blog explains how lease management in Azure Storage works, covering the management of concurrent access to blobs and containers. It discusses key concepts such as acquiring, renewing, changing, releasing, and breaking leases, ensuring only the lease holder can modify or delete a resource for a specified duration. Additionally, it explores common troubleshooting scenarios in Azure Storage Lease Management.

Lease management in Azure Storage allows you to create and manage locks on blobs for write and delete operations. This is particularly useful for ensuring that only one client can write to a blob at a time, preventing conflicts and ensuring data consistency.

Key Concepts:

Lease States: A blob can be in one of several lease states, such as Available, Leased, Expired, Breaking, and Broken. Each state indicates the current status of the lease on the blob.
Lease Duration: The duration of a lease can be between 15 to 60 seconds, or it can be infinite. An infinite lease remains active until it is explicitly released or broken. In versions prior to 2012-02-12, the lock duration is 60 seconds.
Lease Actions: There are several actions you can perform on a lease, including acquiring, renewing, releasing, and breaking the lease.

The diagram below illustrates the five states of a lease and the commands or events that trigger lease state changes. A lease in Azure Blob Storage manages concurrent access to blobs and containers, ensuring that only the lease holder can modify or delete a resource for a specified duration.

Available

The blob is not leased at present and is available for leasing. It transitions to the Leased state when acquired and stays in the Available state when released or written to.

Leased

When a lease is acquired, the blob enters a leased state and is locked from modification by others; it is available when released, leased when renewed or changed, breaking with a break period greater than 0, broken with a break period of 0, and expired when the lease period ends.

Expired

The lease has expired, but it can be renewed or re-acquired, transitioning to leased when acquired or available when terminated.

Breaking

The lease is terminated with a non-zero break period, remaining in the Breaking state if greater than 0, and entering the Broken state once the period ends or is set to 0, becoming available when released.

Broken

When a lease is broken, clients can acquire a new lease; it becomes Leased when acquired, remains broken if broken again, and moves to available when released or written to.

In addition, Lease management in Azure Storage allows you to create and manage locks on containers to control access for delete operations. This is particularly useful for ensuring that only one client can delete a container at a time, preventing conflicts and ensuring data consistency.

Troubleshooting Scenarios

Scenario 1: Lease ID Mismatch:

Error:

"The lease ID specified did not match the lease ID for the blob. RequestId:xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx Time:YYYY-MM-DDTHH:mm:SS.Z

Cause:

This error occurs when:
• The lease ID provided in the request does not match the actual lease ID currently held by the blob.
• The lease has already expired or was broken, making the lease ID invalid.
• Another process has acquired a new lease, making the old lease ID incorrect.

Resolution:

Verify the Correct Lease ID
• Get the correct lease ID before performing lease operations.
• You can retrieve the current lease ID from your application logs or metadata if previously stored.
Acquire a New Lease If Needed
• If the lease has already expired or was broken, you must acquire a new lease before proceeding. To acquire a lease, please refer to lease action command above
Use the Correct Lease ID for Subsequent Operations
• Ensure all lease-related requests (renew, release, break, change) use the correct lease ID that was returned when acquiring the lease.

Scenario 2: Lease ID Missing

Error:

“There is currently a lease on the blob and no lease ID was specified in the request.
RequestId: xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx Time:YYYY-MM-DDTHH:mm:SS.Z

Cause:

This error occurs when:
• You are trying to renew, release, or change a lease but did not include x-ms-lease-id in the request.
• The blob is already leased, but you are trying to acquire a new lease without breaking the existing one first.
• You are attempting to break a lease without including the lease ID when required (some scenarios do not require it)

Resolution:

Include the Lease ID in the Request
• Ensure you provide the correct x-ms-lease-id in the request header.
• Example for renewing a lease. To renew a lease, please refer to lease action command above
Verify If a Lease Exists Before Performing Operations
You can check the lease status by using the Get Blob Properties API:

curl -I -X HEAD "https://mystorageaccount.blob.core.windows.net/mycontainer/myblob?SAS_TOKEN" \ -H "x-ms-version: 2021-08-06"

If the lease status is “unlocked”, then there is no active lease, and you do not need to specify a lease ID.

Scenario 3: Lease ID Not Found:

Error:

There is currently no lease on the blob.
RequestId: xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx
Time: YYYY-MM-DDTHH:MM:SS.ssssssZ

Cause:

The error occurs when:
• You provide an incorrect or expired lease ID in lease operations such as renew, release, or change.
• The lease has already expired or been broken, making the lease ID invalid.
• The blob never had a lease in the first place.
• You are trying to release or renew a lease without specifying a lease ID when required.

Resolution:

Check If the Blob Has an Active Lease
Before performing any lease-related operation, you should verify the lease status using the Get Blob Properties API:

curl -I -X HEAD "https://mystorageaccount.blob.core.windows.net/mycontainer/myblob?SAS_TOKEN" \ -H "x-ms-version: 2021-08-06"

Look for these headers in the response:
x-ms-lease-status: locked → A lease is present.
x-ms-lease-status: unlocked → No lease is currently active.
x-ms-lease-state: expired → The lease has expired.

Ensure You Use the Correct Lease ID
If a lease is present, but you’re getting an error, ensure you are using the correct lease ID from when the lease was acquired. If you don’t have the lease ID, you may need to break the lease and acquire a new one.

If No Active Lease Exists, Acquire a New Lease. If the blob does not have an active lease, you need to acquire one before performing lease-related operations:

If the lease ID is incorrect or missing, check the lease status first.
If no lease exists, acquire a new lease before proceeding.
If the lease ID is lost, break the lease and acquire a new one.

Scenario 4: Upgrade Azure Blob Storage with Azure Data Lake Storage capabilities

Error:

Unable to proceed HnsOn migration due to incompatible feature, Blob has active lease

Cause:

The HNS upgrade fails if any blob/container has an active lease.

Resolution:

The upgrade might fail if an application writes to the storage account during the upgrade. To prevent such write activity:
Quiesce any applications or services that might perform write operations.
Release or break existing leases on containers and blobs in the storage account.
After the upgrade has completed, break the leases you created to resume allowing write access to the containers and blobs.

Note

Breaking an active lease without gracefully disabling applications or virtual machines that are currently accessing those resources could have unexpected results. Be sure to quiesce any current write activities before breaking any current leases.

Additional Resources:

For more detailed information and troubleshooting tips, you can refer to the following resources:

Azure PaaS Blog articles

How to get blob Total Blob Count and Total Capacity with Blob Inventory

Table of Contents

Approach

Introduction to the Blob Inventory Service

Steps to enable inventory report

Support Documentation

Disclaimer

Redis Keys Statistics

Deriving expiry days and remaining retention days for blobs through blob inventory

Exclude Prefix in Azure Storage Action: Smarter Blob Management

Developer Tier APIM + Self-hosted Gateway

Finding the Right Page number in PDFs with AI Search

How to enable alerts in Batch especially when a node is encountering high disk usage

Best practices to follow to avoid issues with high disk usage in Azure Batch:

Now let’s see how to get notified when a Batch node experiences high disk usage.

Converting Page or Append Blobs to Block Blobs with ADF

Rehydrating Archived Blobs via Storage Task Actions

Create a Task :-

Add the Assignment :-

Enable the Task Assignment :-

Monitoring the runs :-

Reference Links :-

Integrating Azure Monitor in Azure Batch to monitor Batch Pool nodes performance

Lifecycle Management of Blobs (Deletion) using Automation Tasks

[AI Search] LockedSPLResourceFound error when deleting AI Search

Introducing Azure SRE Agent

Key Capabilities

Evaluating usage and performance trends

Proactive detection and remediation of security vulnerabilities

Automated incident response and Faster root cause analysis

Incident mitigation

Close the loop with developers

Share your thoughts

Additional resources

Custom Tracing in API Management

How Networking setting of Batch Account impacts simplified communication mode Batch pool

Cause:

Solution:

AI Resilience: Strategies to Keep Your Intelligent App Running at Peak Performance

Stay Online

Architecture Reference

Cost consideration

Conclusion

Streaming and Analyzing Azure Storage Diagnostic Logs via Event Hub using Service Bus Explorer

Prerequisites

Let's take a look at the steps below for configuring Service Bus Explorer and Reviewing the logs ahead

Tips for Migrating Azure Event Hub from Standard to Basic Tier Using Scripts

Automate creation of work items in ADO and Export/Import workflow packages

Lease Management in Azure Storage & Common troubleshooting scenarios