Migrate Data Ingestion from Data Collector to Log Ingestion - Part 2

In Part1, we discussed how to use HTTP action to migrate from the deprecated Data Collector API to Log Ingestion API.

However, since the deprecated Data Collector API was allowing message payload up to 30MB, some migration scenarios encountered issues; since the new Log Ingestion API has only 1MB payload limit. This applies to both HTTP action, and the preview (Log Ingestion connector)

Azure Monitor service limits - Azure Monitor | Microsoft Learn

When sending payload larger than 1MB, the HTTP action would fail with the error:

In order to overcome this, we can chunk the array payload, before calling the Ingestion API. We need to chunk the array payload, so each produced chunk is around 800KB

To do this, you can divide the size of your full payload, by the number of records it has, and this would give an average record size. Then divide 800,000 bytes with the average record size, and this will give you the chunk size.

For example, if your payload size is 1,200,259 bytes, and the number of records is 2500, then the average record size is:
1,200,259 / 2,500 =~ 480 bytes

To find the chunk size, we divide 800,000 bytes by 480 bytes

800,000 / 480 =~ 1666 records.

So if we send around 1600 records, this should be less than 1MB.

To chunk your array payload, we can use the chunk expression inside a compose action.

You can use the expression below for chunking inside the compose action:

After the compose action, add for-each action, and use the output of the compose action as an input:

Inside the for-each, use the normal HTTP to send the logs, and in the body, choose the for-each current item