How to set offset for Event Hubs Trigger
Published Jul 18 2023 08:55 PM 5,020 Views
Microsoft

We understand that when retrieving data from Event Hubs, you can provide the offset to retrieve the data from offset.

The same mechanism applies when using the Function App Event Hubs Trigger, which stores the offset information in a storage account.

In this document, we will discuss how to check the offset information for Event Hubs and how to set or reset the value.

Prerequisite

  • EventHubs with Standard Pricing tier
    • EventHub name : samples-workitems
    • Partition count : 4
    • Retention time : 168 hours

Calvin_Cai_0-1689134581851.png

 

 

  • Function App with EventHubs Trigger

ServiceBusExplorer

Before we connect to EventHubs from function app, we could use ServiceBusExplorer to test send/receive data from EventHubs.
You could download ServiceBusExplorer-4.1.112  version to do the test.

  1. Navigate EventHubs instance and click "Shared access policies" -> "RootManageSharedAccessKey"

    Calvin_Cai_1-1689134613137.png

     


     

  2. Open Service Bus Explorer we just downloaded, click "File" and select "Connect", select "Enter connection string" and paste the connection string.

    Calvin_Cai_2-1689134644963.png

     


     

  3. Once you connected to EventHubs, you will see the entity name, here we already created "samples-workitems" for testing

    Calvin_Cai_19-1689133897969.png

     

  4. Right click "samples-workitems" and click "Send Events" menu

    Calvin_Cai_20-1689133907877.png

     

  5. Fill the Message Text and click "Start", now you sent a message to EventHubs

    Calvin_Cai_3-1689134669592.png

     


     

  6. Right click a Consumer Groups, like "$Default", click "Create Consumer Group Listener"

    Calvin_Cai_22-1689133931145.png

     

  7. Click Start button, and it will receive all the Events, you could see the Offset(it could be a large number) and SequenceNumber value
    And you could close the window and create listener again, you could receive the messages again.

    Calvin_Cai_23-1689133941644.png

     

Configure Function App to receive data from EventHubs

Here we use a PowerShell function to test receive data from EventHubs, please refer below sample file

  • function.json
{
  "bindings": [
    {
      "type": "eventHubTrigger",
      "name": "eventHubMessages",
      "direction": "in",
      "eventHubName": "samples-workitems",
      "connection": "EVENTHUB_CONN",
      "cardinality": "many",
      "consumerGroup": "$Default"
    }
  ]
}
  • run.ps1
param($eventHubMessages, $TriggerMetadata)

Write-Host "PowerShell eventhub trigger function called for message array: $eventHubMessages"

$json = ConvertTo-Json $eventHubMessages
Write-Host $json

$json = ConvertTo-Json $TriggerMetadata
Write-Host $json

$eventHubMessages | ForEach-Object { Write-Host "Processed message: $_" }

 

The Configuration EVENTHUB_CONN configred the connection string to EventHubs
After the code is deployed, function app should able to process data from EventHubs, you should able to see below messages

Calvin_Cai_4-1689134723994.png

 

 

 

 

Check The offset information

  1. Open function app from Azure Portal and navigate to Configuration Page, check the "AzureWebJobStorage" value and find storage account name
    Calvin_Cai_6-1689134772404.png

     

  2. Navigate to the storage account instance and you would see "azure-webjob-eventhub" container
    Calvin_Cai_7-1689134797539.png

     


     

  3. Navigate to <eventhub instance>/<eventhub entity>/<consume group>/checkpoint folder, and you would see partitions file
    Calvin_Cai_27-1689134009537.png

     

  4. Click the partition file and you would see the Metadata stored "offset" and "sequencenumber" information
    Calvin_Cai_28-1689134037293.png

     

  5. You could update the offset and sequencenumber information and save it, Or you could just delete the partition file
  6. The function app will process the existing message accordingly (from the new offset, or from begining)

Refrences

1 Comment
Co-Authors
Version history
Last update:
‎Jul 13 2023 12:51 AM
Updated by: