Event hub -message batch size limitation in JavaScript

Published Jul 29 2021 01:19 AM 1,995 Views
Microsoft

Recently, we had a customer who wants to test the Event Hub quota in JavaScript. Then I did some tests about add Event Hub events in Batch through JavaScript SDK.

This blog will explain my tests send EventHub events in batch through JavaScript SDK.

Pre-requirements:

Before we start please read these documents What is Azure Event Hubs , Quotas and limits - Azure Event Hubs and Send or receive events from Azure Event Hubs using JavaScript

From these documents, we learn the followings:

Event Hub: It’s a big data streaming application, it can handle millions of events pre second.

No mater events individually or batched they have same maximum threshold. I would be rejected if reach to the limit. Basic pricing tier is 256 KB, and Standard, Premium and Dedicated tier are 1MB.

JavaScript enable both send and receive function in batch.  There is a function “tryAdd” to add event to Batch.

If the event in batch reach to 1MB, do you think it will pop out  any exceptions? Let’s test about it.

Tests:

In below code, I tried to add 100000 events in batch and send to my Event Hub. Each event uses same parameter with 1000 characters.

 async function main() {

 

  // Create a producer client to send events to the event hub.

  const producer = new EventHubProducerClient(connectionString, eventHubName);

 

  var count=0

  // Prepare a batch of three events.

  const batch = await producer.createBatch();

  //var data=genData(100000)

  for (var i = 0; i < 10000; ++i) {

  var event= 'longcharacters’

  batch.tryAdd(event)

  count=i

  } 

 

  var milliseconds = new Date().getTime();

  // Send the batch to the event hub.

  await producer.sendBatch(batch);

  var milliseconds2 = new Date().getTime();

  // Close the producer client.

  //await producer.close();

  console.log("A batch of"+ count+"events have been sent to the event hub from "+milliseconds+ " end "+milliseconds2);

}

main().catch((err) => {

  console.log("Error occurred: ", err);

});

Then I found this program didn’t send all the 100000 events. From the metric chart I know that 2.57 K sent to Event Hub. And the incoming size above 1MB. But there isn’t any exception pop out in the debug mode.  

Scarlett_liu_0-1627545764828.png

 

From this document azure-sdk-for-js/sdk/eventhub, we know that “tryAdd” would return “false” to indicate that no more events can be added to the batch of event using sendBatch.

To validate it, I updated this part of code in this way:

 //var data=genData(100000)

  for (var i = 0; i < 100000; ++i) {

  var event= 'longcharacters’'

  let test=batch.tryAdd(event)

  if (test){

    count=i

    console.log('---- add completed'+i)

  }

  else{

    console.log("the limit of batch"+count);

  }

 

The debug log shows that after added 25573 events to batch, “tryAdd” return to “false”

 

Scarlett_liu_1-1627545764832.png

 

 

So that means if the batch reach to maximum threshold, “tryAdd” would not add new event to batch any more. Then we can use this Boolean value to evaluate the event total size.  Adding to another batch or separate to more batches are work arounds for this scenario.

 

To sum up, this test is to validate any exception will pop out if events add in one batch above maximum threshold in JavaScript. The function “tryAdd” it would return “false” value then stop to add event to the bach. We can use this value to handle the situation when total size of events above limit in your program.

Co-Authors
Version history
Last update:
‎Jul 29 2021 06:44 PM
Updated by: