Service Bus - duplicate detection for partitioned messaging entity

Pre-requirements:

Before we start, please read these document about Duplicate Detection, Partitioned queues and topics and Message Sessions 

 

From the above Pre-requirements, we learn the followings

• Enabling Duplicate detection helps to ensure that a duplicate message, which has the same message Id cannot be sent into a messaging entity during a specified time.

 

• Service Bus Partitions enable queues and topics or message entities to be partitioned across multiple message brokers and messaging stores. Enable partitioning the overall throughput will separate to different partition. Partition key can be used in some scenarios, such as sessions or transactions, require messages to be stored in a specific partition.

 

 

• Microsoft Service Bus Session enable joint and ordered handling of unbounded sequences of messages.  There are two patterns of it , FIFO and request-response pattern. Any sender can create a session when submitting messages into a topic or queue by setting the SessionId property to some application-defined identifier that is unique to the session.

 

So, from these above knowledges, we know that Azure Service Bus Queue and Topic/Subscription can enable for Duplicate Detection, Partitions and Sessions. But what’s the relationship between them?

From the meaning of Duplication detection, we know that there is a MessageId of each message. Definition of MessageId is shown below

“If the queue or topic has the RequiresDuplicateDetection property set to true and the SessionId or PartitionKey properties are not set, then the MessageId property value serves as the partition key”

 

This means the duplicate detection only can work when SessionID or PartitionKey are not be set. Is that a correct statement? Let’s do a test!

 

Test Entities:

 

Queues:

There are three queue I used, testqueue, testqueue2, testqueue3. At first time all of them have 0 messages.

• testqueue:  Enable Batched Operation, Requires Duplicate Detection. And set duplicate detection for 10 mins.

 

• testqueue2: Enable Batched Operation, Requires Duplicate Detection, Requires Session

 

• testqueue3: Enable Batched Operation, Requires Duplicate Detection, Requires Session, Enable Partition

 

 

My Program:

Here is the code. The different in this program used in each time, just the BrokeredMessage SessionID and PartitionKey that in red color.

The Session ID here set with a random value. From the above document, when enable both Partition key and Session ID, the value of them need to be same. Otherwise Service Bus returns an invalid operation exception.

 

    class Program

    {

        static string connectionString = [ConnectionString];

        static string queueName = "testqueue3";// testqueue,testqueue2

        static void Main(string[] args)

        {

            MainAsync();

        }

        static void MainAsync()

        {

            QueueClient sendClient = QueueClient.CreateFromConnectionString(connectionString, queueName);

            //create a sender on the queue

 

            var session = Guid.NewGuid().ToString();

            var partitionKey = session;

            sendClient.SendBatch(Enumerable.Range(0, 10).Select(n =>

            {

 

                Console.WriteLine("test my message n{0}", session);

                BrokeredMessage test = new BrokeredMessage("Hello World!") { SessionId = session, , PartitionKey = session, MessageId = "deadbeef-dead-beef-dead-beef" + n };

 

                return test;

            }

            ));

        }

}

Test Round 1:

To find the message, I sent 10 message each time first.

• testqueue (Enable Batched Operation, Requires Duplicate Detection):  send 10 messages first, then immediately  send another same 10 messages, it still displays 10 messages.

• testqueue2(Enable Batched Operation, Requires Duplicate Detection, Requires Session): send 10 messages first, and send another same 10 messages, here also display 10 messages.

 

• testqueue3(Enable Batched Operation, Requires Duplicate Detection, Requires Session, Enable Partition): Add new message twice the message count turn to 20.

You can see in the list there are message with duplicate MessageId.

 

 

Then I checked for same MessageId “deadbeef-dead-beef-dead-beef0” , and I found they have  different Partition Keys.

 

 

Here are these three queues result in this test.

 

The above test about messages sent in duplicate detection duration. How about we wait for 10 mins to check after the duplicate detection time?

 

 

 

Test Round 2

 

After 10 mins, I sent 1000 messages again twice. The result shows, testqueue and testqueue2 have 1000 included the 10 messages sent before. For testqueue3, there are 2020 messages.

 

 

Test Result Summary

• From the first round test result, it indicates that messages with same MessageId can be detected as duplication in a non-partitioned entity within specified duplicate detection duration (10 mins).

 

• However with partitioned entity, duplicate messages which has same MessageId may not be detected as duplication due to they can end up in a different partition. In conclusion, messages can only be detected as duplication within the specified duplicate detection duration when they are landed in the same partition, i.e. same partition key

 

 

• For the second test with 1000 message were sent twice, it indicates that the duplicate duration detect only works for the specified duration. After the duplicate detection period, the Messages with same MessageId can be sent to the same Service Bus messaging entity.