Load Testing Azure Event Hubs services with restricted public access

[flecoqui]

Load Testing Azure Event Hubs services with restricted public access

This article describes how to use Azure Load Testing to test a service based on Azure Event Hubs with a restricted public endpoint. The access to the Azure Event Hubs endpoint is restricted to specific client IP addresses. For instance, the service collects events from different on-premises events sources, analyzes those events and generates alerts as anomalies are detected.

Scenario details

For this scenario, Azure Load Testing is used to test the performances of an architecture based on Azure Event Hubs. Azure Load Testing can be easily configured to support the network requirements of  the targeted service. The Load Testing resources are only deployed during the tests and can be managed from an Azure DevOps pipeline or a Github Action.

Potential use cases

Organizations can use Azure Load Testing to test performances of the following use cases based on Azure Event Hubs:

• Anomaly detection (fraud/outliers)
• Application logging
• Analytics pipelines, such as Clickstream
• Live dashboards
• Archiving data
• Transaction processing
• User telemetry processing
• Device telemetry streaming

Architecture of the system we want to load test

Dataflow

This paragraph describes a possible sample scenario of the system we want to load test. First the Azure Event Hubs namespace firewall has been configured to accept connections from the different sources on Internet adding the source IP addresses in the firewall configuration.

1. The Azure Event Hubs collects the events from the different sources

2. The service analyzes the events using different possible resources like Azure Data Factory, Azure Machine Learning, Azure Stream Analytics, Azure Synapse Analytics. If an anomaly is detected, the alert is sent to the Output Event Hub.

3. The alert can trigger the transmission of an e-mail, notification. Moreover, the alert can be stored in Azure Storage and/or Azure SQL Server.

Load testing Architecture

The load testing of this infrastructure requires the deployment of Azure Load Testing service using this scenario to load test a public endpoint with access restrictions.

In that case, beyond the Azure Load Testing service, an Azure Virtual Network and an Azure NAT Gateway are deployed to use the same source public IP address for the requests coming from the load testing service. The pipeline (Azure DevOps pipeline, Github Action) running the load test adds the source public IP address in the Event Hub firewall configuration before running the tests and removes it once the tests are completed.

Load testing scenario

This paragraph describes all the steps to load test Azure Event Hubs with restricted public access.

1. Configure the Azure Event Hubs namespace firewall to accept connections from the different sources on Internet including the Azure Load Testing service adding the source IP addresses in the firewall configuration
2. Launch the load testing service to send requests to the input Event Hub.
3. The Azure Event Hubs collects the events from the different source.
4. The service analyzes the events using different possible resources like Azure Data Factory, Azure Machine Learning, Azure Stream Analytics, Azure Synapse Analytics. If an anomaly is detected, the alert is sent to the Output Event Hub.
5. The alert can trigger the transmission of an mail, notification. Moreover, the alert can be stored in Azure Storage and Azure SQL Server.
6. While the load testing is still sending requests to the Azure Event Hub, the load testing pipeline sends events that should trigger an alert, the same pipeline reads events (alerts) from the output event hub to measure the latency of the system.
7. When the load testing duration is reached, the service stops sending requests to the input Event Hub. The load testing results are successful if the alert latency is not degraded with the high ingress traffic.

Components

Beyond Azure Load Testing and Azure Event Hubs, this paragraph describes the list of services which could be deployed on Azure to analyze the incoming events and possibly to generate an alert when an anomaly is detected.

• Azure Load Testing: Enable developers and testers to generate high-scale load and run simulations that reveal actionable insights into app performance, scalability, and capacity with a fully managed load-testing service.
• Create tests quickly without prior knowledge of load testing tools, or upload your existing Apache JMeter scripts. Gain specialized recommendations backed by comprehensive metrics and analytics, and support continuous improvement through automated continuous integration and continuous delivery (CI/CD) workflows—all with a testing service built for Azure.
• Azure Event Hubs: Stream millions of events per second from any source to build dynamic data pipelines and immediately respond to business challenges.
• Azure Machine Learning: Empowers data scientists and developers to build, deploy, and manage high-quality models faster and with confidence.
• Azure Data Factory: Integrate all your data with Azure Data Factory—a fully managed, serverless data integration service.
• Azure Synapse Analytics: Limitless analytics service that brings together data integration, enterprise data warehousing, and big data analytics.
• Azure Stream Analytics: Go from zero to production in minutes using SQL—easily extensible with custom code and built-in machine learning capabilities for more advanced scenarios.
• Azure Kubernetes Service: The quickest way to start developing and deploying cloud-native apps in Azure, data centers, or at the edge with built-in code-to-cloud pipelines and guardrails.
• Azure Blob Storage: A massively scalable object storage for any type of unstructured data, including images, videos, audio, documents, and more. It's easy and cost effective.
• Azure SQL Database: Fully managed relational database service built for the cloud. Build your next app with the simplicity and flexibility of a multi-model database that scales to meet demand.
• Email: Create automated task and workflows with Azure Logic Apps and Microsoft 365 Outlook Connector to send an email.

Considerations

These considerations implement the pillars of the Azure Well-Architected Framework, which is a set of guiding tenets that can be used to improve the quality of a workload. For more information, see Microsoft Azure Well-Architected Framework.

Cost optimization

Cost optimization is about looking at ways to reduce unnecessary expenses and improve operational efficiencies. For more information, see Overview of the cost optimization pillar.

Use the Azure pricing calculator  to get customized pricing estimates.

Next steps

- Get started with this  Azure sample in load testing Azure Event Hubs services with restricted public access 
- A sample implementation load testing an Eventhubs based infrastructure from either a dev container in Visual Studio Code or an Azure DevOps pipeline or a Github Action is available here