How to start Azure queue processing by multiple WebJob instances when time constant expired or queue

[Andrei_R]

There is a WebJob (many instances may be) and Azure queue as an input, InputQueue. There is another temporary Azure queue, named BatchQueue. In InputQueue we have 1 object per message. WebJob must put objects from InputQueue to BatchQueue, one by one. Then WebJob must process BatchQueue. WebJob can check BatchQueue every time when WebJob puts new object in BatchQueue.

Processing of the BatchQueue is: get ALL objects from queue and pack it in single file and then cleanup BatchQueue.

I need to start BatchQueue processing when:

1. BatchQueue length > MaxLength OR
2. Time interval between last processing and Now() is more then TimeInterval

I think to save text file with time of last processing in the blob storage and rewrite it during every processing. It seems to be a bit tricky and strange.

I'm trying to find some another "out of the box" tool in the Azure for it... Could somebody point to some safe way? I mean when we have multiple instances of our WebJob.

[#2]

Hello @Andrei_R ,

To ensure safe processing of BatchQueue when you have multiple instances of your WebJob, you can use Azure Functions to schedule the processing. Here are the steps you can follow:

Create an Azure Function that is triggered by a timer. Set the interval of the timer to the value of the TimeInterval you mentioned.

In the Azure Function, check the length of the BatchQueue. If the length is greater than MaxLength, trigger the processing of the BatchQueue.

To ensure safe processing when you have multiple instances of your WebJob, you can use a distributed lock service like Azure Redis Cache. You can acquire a lock before processing the BatchQueue to ensure that only one instance of the WebJob is processing the BatchQueue at any given time.

After processing the BatchQueue, release the lock.

By using Azure Functions and a distributed lock service like Azure Redis Cache, you can ensure safe processing of the BatchQueue when you have multiple instances of your WebJob.

I hope this helps! Let me know if you have any further questions.

Kind regards,

Luke Madden