Azure Stream Analytics (ASA) uses a SQL query language that has been augmented with powerful temporal operators to analyze data streams. One way to optimize a Stream Analytics job’s performance is to leverage parallelism in query. A parallel job divides the workload of queries into multiple streaming nodes and reduces the overall execution time.
We strongly recommend that you use the Azure Stream Analytics Tools extension in Visual Studio Code (VS Code) for testing and debugging queries locally. For a job to be parallel, you need to align partition keys between all inputs, query steps, and outputs. The Job Diagram Simulator in VS Code allows you to simulate a job running topology with different streaming units (SUs) and provides suggestions for improving your query parallelism.
If you are new to the ASA Tools extension for VS Code, follow this guide to install.
How does the Job Diagram Simulator help?
The Job Diagram Simulator provides a list of features that would help you improve your query parallelism.
- Simulate job running topology. By dragging the Streaming Units slider, the diagram shows how many Partition IDs are allocated to each streaming node. This gives you an idea of how many SUs you need for your job to be parallel.
- Identify which query steps are not partitioned correctly. For example, if the partition keys are not aligned between the input and output, the Job Diagram Simulator will highlight the query step that causes a non-parallel job.
- Improve your query based on the recommendations. Under the Enhancements tab, it provides recommendations for editing your job’s configurations and/or queries to make your job parallel.
If you want to learn more about the Job Diagram Simulator, please visit Optimize query performance using job diagram simulator.