Event Hubs provides partitions to scale consumers for parallel processing. The concept of partitions belongs to Topics under Event Hubs namespace. Topics helps categorize the incoming messages and consumer in a group processes the events from one of the Topic partitions.
When a Topic is created, the number of partitions are specified at the time of creation. For some special cases though, you may have to add partitions after the Topic has been created. This requires you to dynamically accommodate the addition of partitions. This blog describes the behavior of adding partitions to an existing Topic with Event Hubs. Dynamic additions of partitions is available only on Dedicated Event Hubs clusters and not on Standard Event Hubs namespace.
When you add a partition to an existing even hub the event hub client receives a “MessagingException” from the service informing the clients that entity metadata (here entity, is your event hub and metadata is the partition information) has been altered. The clients then will automatically re-open the AMQP links which would have then picked up the changed metadata information. The clients then operate normally.
Kafka clients that run on Event Hubs with the Apache Kafka protocol behave differently from event hub clients that use AMQP protocol. Kafka clients, update their metadata once every ‘metadata.max.age.ms’ milliseconds which can be specified the client configurations. The librdkafka libraries will also use the same configuration. Metadata updates inform the clients of service changes including the partition count increases.
Kafka sender/producer client – Producer always dictates that send requests contain the partition destination for each set of produced records. Thus, all produce partitioning is done client-side with producer’s view of broker metadata. Once the newly added partitions are added to the producer’s metadata view, they will be available for producer requests.
Kafka consumer/receiver client – When a consumer group member performs a metadata refresh and picks the newly created partitions, that member initiates a group rebalance. Consumer metadata then will be refreshed for all group members, and the new partitions will be assigned by the allotted rebalance leader.
Partition can be added only to topics or event hub instances that are in Dedicated Event Hubs clusters. You can only scale up the partitions and once added, you cannot scale them down.
You must be a registered user to add a comment. If you've already registered, sign in. Otherwise, register and sign in.