Overview:
One of the common design patterns in enterprise integration is processing transactions in Sequence. We are pretty much aware of how to implement sequential convoy in BizTalk which behind the scenes gets controlled and managed by BizTalk SQL Db's to serve sequence mechanism.
Similarly, we could use Azure enterprise messaging service - Service Bus to leverage this mechanism using Logic App workflow.
In this blog, we will implement sequential convoy with Service Bus Queue. The same can be replicated with Topic - Subscription as well.
Implementation:
- First, we need to create Service Bus entity-Queue enabled with sessions . Note, the sessions enabled Queues won't support regular messaging process which means, each message should have Session Id. Brief explanation on session feature below.
Service Bus namespace with Queue enabled for sessions - In service Bus , sessions guarantee that the messages will be delivered in order how they received in Queue/Topic Subscriptions. Of course , you might have seen the conventional queue itself serving messages in an order without sessions but it's not guaranteed the FIFO, as messages will be assigned to messaging brokers in Service Bus in round Robbin fashion while receiving.
- Now, we see how to create a Logic App .You can use either the template available in Logic App designer -Correlated In-order delivery Service Bus messages. However, here I am creating it from scratch.
Logic App Workflow definition:
- We use Service Bus trigger - When a message received in a queue and set the Session Id set to Next Available which creates the session with the current message.
- You can implement your business process to be done on received message and use action -"Complete message in a queue"to delete the message from Queue in the current session.
- Now, use until loop to fetch all subsequent messages available in current Session to process. So here, the developer should determine the condition when to terminate the loop either based on count of messages or any string that identifies the last message in a session to suffice the condition to end iterations. In this scenario, I have used message count just as an example to fetch and process 175 messages.
- Inside loop, use action- "Receive a message from a queue" to receive next message in the current session.
- Implement your message process and use complete message in queue action to delete the message from the Queue.
- Increment counter- variable
- So now, we have implemented to receive all messages in the current session (here 175 as counter) and processed those.
- Close the session outside the loop - we always need to close the session once all the messages processed in the current session as a best practice. As the Service Bus behaves unexpectedly when it exceeds the limitation of having 1500 sessions in an entity.
- Finally , Logic App looks like below.
Challenges in real-time: As we are using peek lock mode , the message will be locked till message lock duration time set on the Queue/topic subscription which can be maximum of 5 minutes. When it exceeds this time duration, the lock will be lost on the message and results in duplicate message processing. However, this can be addressed if end system has the duplicate handling mechanism.
So this is quite possible, in real-time integrations message processing may take more than the lock duration time and cause failures while completing a message. In order to handle this, we can use renew message lock pattern to renew the lock before it expires to support the additional time to process message without loosing the lock on message.
We have two actions in service bus to renew the message lock.
- Renew lock on a message - Explicitly renew's lock on message, we use this on conventional entity.
- Renew lock on session: It renews lock on all messages in a session which will be used in session based messages.
Renew lock pattern in Sequential convoy in Logic App:
If we need to implement extensive renew lock pattern where we are not sure how much time the message processing takes and need an auto renew lock mechanism. You could check below workflow which has an extension to the implementation above controlled by variables on auto renew lock.
You may need to implement the exception handling like dead-lettering the messaging on top of this Logic App.
