Azure Event Hubs is a fully managed, multi-protocol, real-time data ingestion service that is designed to serve demanding data streaming workloads using Apache Kafka, AMQP or HTTPs protocols.
Today, we are announcing the general availability of Azure Event Dedicated Self-Serve clusters which are designed for mission-critical Kafka and AMQP workloads that require low-latency and high-volume data streaming with dynamic scaling.
Here are some of the key benefits you will receive from using Event Hubs self-serve scalable clusters for your business-critical data streaming needs.
Ultra-low latency event streaming
Event Hubs dedicated clusters are optimized for low latency event streaming. As Event Hubs self-serve dedicated clusters are based on a new two-tier log storage engine architecture and run on state-of-the-art infrastructure, these clusters provide consistent low latency for most of the event streaming workloads.
The following graph shows the latency numbers we obtained for different event streaming workloads that use Apache Kafka API of Event Hubs Dedicated clusters. As you can see, we consistently maintained an average latency of approximately 10ms, with a P99 latency of around 30ms and a P999 latency of approximately 50ms.
This test is carried out using the Linux Foundation project, OpenMessaging Benchmark Framework which is commonly used for benchmarking message brokers and event streaming services. For this latency test, we used 1 CU self-serve dedicated cluster, 1 event hub/Kafka topic with 100 partitions. Kafka client configuration uses linger duration of 1ms, batch size of 131072, 1 producer and 100 consumer instances.
It's crucial to keep in mind that the performance of a dedicated clusters can greatly vary depending on the specific use case. A multitude of factors can influence the overall performance, such as the number of producers and consumers, the number of partitions, the producer and consumer configurations, the payload and batch size, and the egress rate. To ensure accurate performance results, it's best to test the clusters against your specific workload.
Streaming large volumes of data
Dedicated clusters can stream events at gigabytes per second or millions of events per second scale for the majority of the use cases. Also, these clusters can be easily scaled to accommodate changes in event streaming volume.
Consistent performance
Event Hubs self-serve Dedicated clusters ensure consistent performance with guaranteed capacity. As these clusters are single tenant, there is no cross-tenant interference and the new log storage engine architecture, and the updated infrastructure ensures consistent performance.
Self-serve scaling
With the new Dedicated clusters, you no longer need to create a support request to scale in or scale out your clusters. Now, you can effortlessly scale it with just a few clicks using the Event Hubs portal.
Built on top State-of-the-Art Hardware and Software Infrastructure.
Event Hubs Dedicated self-serve clusters are powered by cutting-edge hardware and software technology, resulting in improved performance compared to its previous generation. These clusters can deliver superior performance when compared to the same number of scale units (CUs) from the previous generation.
Two tier-log storage engine architecture for better performance
Event Hubs Dedicated self-serve clusters introduce a brand-new, two-tier, native-code log storage engine that provides predictable and much lower ingestion and end-to-end latencies without any durability compromises. This enables you to ingest and processes large volumes of events and data with high throughput, low latency, and high reliability.
This is the same storage engine that has been battle-tested in our Premium tier and this makes Event Hubs dedicated clusters more powerful.
Single Tenancy
With a dedicated Event Hubs cluster, you are the only tenant using the gateways and brokers in your cluster. Although the multi-tenant versions (Standard and Premium tiers) are highly secure, there may be specific enterprise requirements that can only be met with a single-tenant service solution.
Cost effective – Do more with less scale units.
New Event Hubs Dedicated clusters can bring you cost savings as you may need fewer Capacity Units allocated to your Dedicated clusters. Also, since we support self-serve scaling you can scale in or scale out cluster capacity to suit your needs which can reduce the overall monthly cost you pay for your dedicated cluster.
Get Started Today
Unlock the full potential of your streaming data with Azure Event Hubs Dedicated Self-Serve Scalable Clusters. Here's how to get started:
- Creating a self-serve scalable Dedicated cluster.
- To Learn more about Event Hubs Dedicated clusters see Overview of Azure Event Hubs dedicated tier - Azure Event Hubs | Microsoft Learn.
Microsoft
