This article guides you on configuring the Event Hub Trigger of Logic App and later automate the workflows for the events pulled from Event Hub.
To introduce about the Event Hub Trigger :
All Event Hub triggers are long-polling triggers, which means that the trigger processes all the events and then waits 30 seconds per partition for more events to appear in your Event Hub. So, if the trigger is set up with four partitions, this delay might take up to two minutes before the trigger finishes polling all the partitions. If no events are received within this delay, the trigger run is skipped. Otherwise, the trigger continues reading events until your Event Hub is empty. The next trigger poll happens based on the recurrence interval that you specify in the trigger's properties.
- The Event Hub Trigger pulls all the events until the number reaches the "Maximum event Counts "from one partition at a time.
- The next poll for next partition happens after 30 seconds gap.
- If next partition doesn't have any data the trigger would count as skipped and again we have delay of 30 seconds.
- As a whole we would not know how long the Event Hub trigger would take to complete to poll all the partitions.
It depends on the previous poll time.
Hence, we would not be able to poll all the partitions of Event Hub at the same time by targeting all the partitions by LA Trigger as shown below.
If we configure the Event Hub Trigger without "Minimum Partition key" and "Maximum Partition key", then by default all partitions will be polled.
If we configure the Event Hub Trigger with "Minimum Partition key" and "Maximum Partition key"(considering the number of partitions), then only specific partitions will be polled. In this case : partition 0,1,2 and 3 will be polled.
In either of the above cases we will have a Single logic App trigger polling for all the partitions with a polling interval of 3 minutes.
Suppose all the Event source data is pushed into a single partition (say partition 3), then it will take more than 12 minutes to pull the data.
Note : If the data is pushed into the Event Hub without targeting any of partitions then we cannot assume the data is distributed to all the partitions equally.
In a real time scenario, we may require to process all the events from all the partitions of Event Hub at the same time.
For this instance we shall again keep an Event Hub with only 4 partitions for demo.
Since we know a single Logic App Event Hub trigger shall poll all the partitions in round robin fashion and each trigger shall poll each partitions and our used case will not be satisfied. Hence, we shall go for 4 different Logic App which points to each Event Hub partition.
Each Logic App can point to different Partitions as shown in the below image.
If we have the common Logic of processing the Events then we can opt for Common Logic App which does the actual Processing of the Events. These 4 Logic App can pull the data and call the Common Logic App for processing. This helps in LA definition management and also helps in reducing the complexity.
Later we can call the Common Logic app from all the Other Logic Apps.
The above design can be made scalable in various aspects based on number of Events and Event Hub Partitions.
For example :
Let's consider the scenario where you have more Events that are published and you will have to add more partitions to the Event hub. For now we keep it as 8 partitions.
If we have a single Logic App to poll all the partitions from 0 to 7 , it would take 24 minutes to poll all the data if we have configured to poll the Event Hub for every 3 minutes with total of 8 triggers fired.
Similarly, if we have the polling frequency set for 1 minute it would take at least 8 minutes to complete to poll all the partitions.
Trigger History for reference :
So by the time the Trigger would poll the partition which has more data the chances of overloading the EventHub would be even more because the events are not pulled from Event Hub in the rate that is received from the partition.
In Such scenarios we can add few more Logic Apps where each LA shall poll less partitions which in turn increases the frequency of poll.
For ex: we can clone 3 more Logic Apps separating the main logic of processing/automation of the Events in Child Logic Apps.
Choose Clone option under Overview of Logic Apps and create 3 more Logic Apps.
Later change the properties of LogicApp to poll 2 partitions each. For ex Logic App-1 to partition 0 and 1, Logic App-2 to partition 2 and 3, Logic App-3 to Partition 4 and 5 , Logic App-4 to partition 6 and 7.
Since the major logic goes into the Child(Common Logic App) there would be drastic reduction in maintenance work. If any changes related to business can be done in Child Logic App.
Reference : https://docs.microsoft.com/en-us/azure/connectors/connectors-create-api-azure-event-hubs