Hubs, Sharding and Partitioning
It seems there is a some misunderstanding related to sharding and partitioning when talking about IoT and Microsoft Azure platform. Here is my recommendation about how we should thnk about this. EventHub uses internally messaging feature called 'partitioning'. To understand how that works I would recommend to read more about this in context of service bus. It basically enables multiple message brokers instead of a single one. So, this is called partitioing even if some people might call this shrading. IotHub also uses same (or similar) technic as EventHub. However, when we talk about IotHub 'shards' we do not think about partitioning on message level. It is rather using of multiple IotHubs inside of same solution. This is not some specific technology under the hub. To use multiple IotHubs inside of a single solution, You should decide how device provisioning is done. Dev1-Hub1, Dev2->Hub2 etc.Processing ordered IoT streams in Azure
There are thousands of articles on the Internet talking about how the cloud can be used to process discrete (un-ordered) streaming IoT data. But almost all are short on specifics and the design trade-offs you need to make when processing ordered IoT streams, at scale. This article discusses why scale can be difficult to achieve, especially when the ordering of the events (aka Series events) is important. Jeff Hollan wrote an article which illustrates the challenge when using Azure Functions, for example. And if you need a primer on Real-time processing, Events, Messages, Series events and Discrete events, please review the following articles: Real time processing Events, Data Points, and Messages - Choosing the right Azure messaging service for your data Choose between Azure messaging services - Event Grid, Event Hubs, and Service Bus
New Akka Streams and Apache Flink tutorials/samples for Event Hubs for Apache Kafka Ecosystems
First published on on Jun 13, 2018 Introducing two new Event Hubs for Apache Kafka Ecosystems tutorials - Akka Streams and Apache Flink!Event Hubs for Kafka Ecosystems combines the power and simplicity of Event Hubs with the wide ecosystem of Apache Kafka.Dynamically adding partitions to a topic/event hub under Event Hubs namespace
Event Hubs provides partitions to scale consumers for parallel processing. The concept of partitions belongs to Topics under Event Hubs namespace. Topics helps categorize the incoming messages and consumer in a group processes the events from one of the Topic partitions. When a Topic is created, the number of partitions are specified at the time of creation. For some special cases though, you may have to add partitions after the Topic has been created. This requires you to dynamically accommodate the addition of partitions. This blog describes the behavior of adding partitions to an existing Topic with Event Hubs. Dynamic additions of partitions is available only on Dedicated Event Hubs clusters and not on Standard Event Hubs namespace.
