How to set offset for Event Hubs Trigger

Microsoft

Jul 19, 2023

We understand that when retrieving data from Event Hubs, you can provide the offset to retrieve the data from offset.

The same mechanism applies when using the Function App Event Hubs Trigger, which stores the offset information in a storage account.

In this document, we will discuss how to check the offset information for Event Hubs and how to set or reset the value.

Prerequisite

EventHubs with Standard Pricing tier
- EventHub name : samples-workitems
- Partition count : 4
- Retention time : 168 hours

Function App with EventHubs Trigger

ServiceBusExplorer

Before we connect to EventHubs from function app, we could use ServiceBusExplorer to test send/receive data from EventHubs.
You could download ServiceBusExplorer-4.1.112 version to do the test.

Navigate EventHubs instance and click "Shared access policies" -> "RootManageSharedAccessKey"
Open Service Bus Explorer we just downloaded, click "File" and select "Connect", select "Enter connection string" and paste the connection string.
Once you connected to EventHubs, you will see the entity name, here we already created "samples-workitems" for testing
Right click "samples-workitems" and click "Send Events" menu
Fill the Message Text and click "Start", now you sent a message to EventHubs
Right click a Consumer Groups, like "$Default", click "Create Consumer Group Listener"
Click Start button, and it will receive all the Events, you could see the Offset(it could be a large number) and SequenceNumber value
And you could close the window and create listener again, you could receive the messages again.

Configure Function App to receive data from EventHubs

Here we use a PowerShell function to test receive data from EventHubs, please refer below sample file

function.json

{
  "bindings": [
    {
      "type": "eventHubTrigger",
      "name": "eventHubMessages",
      "direction": "in",
      "eventHubName": "samples-workitems",
      "connection": "EVENTHUB_CONN",
      "cardinality": "many",
      "consumerGroup": "$Default"
    }
  ]
}

run.ps1

param($eventHubMessages, $TriggerMetadata)

Write-Host "PowerShell eventhub trigger function called for message array: $eventHubMessages"

$json = ConvertTo-Json $eventHubMessages
Write-Host $json

$json = ConvertTo-Json $TriggerMetadata
Write-Host $json

$eventHubMessages | ForEach-Object { Write-Host "Processed message: $_" }

The Configuration EVENTHUB_CONN configred the connection string to EventHubs
After the code is deployed, function app should able to process data from EventHubs, you should able to see below messages

Check The offset information

Open function app from Azure Portal and navigate to Configuration Page, check the "AzureWebJobStorage" value and find storage account name
Navigate to the storage account instance and you would see "azure-webjob-eventhub" container
Navigate to <eventhub instance>/<eventhub entity>/<consume group>/checkpoint folder, and you would see partitions file
Click the partition file and you would see the Metadata stored "offset" and "sequencenumber" information
You could update the offset and sequencenumber information and save it, Or you could just delete the partition file
The function app will process the existing message accordingly (from the new offset, or from begining)