Azure Stream Analytics is a fully managed, real-time analytics engine designed to analyze and process fast moving streams of data. It allows developers to use SQL to define a rich set of queries enabling companies to analyze, filter, or otherwise conduct real-time processing of unbounded datasets.
Streaming Units (SUs) represent the computing resources that are designated to run the Stream Analytics job. For example, let's say you create a job and allocate 3 SUs, the configuration of 3 SUs in this instance determines how much memory and compute power is allocated to that job. The logic behind this is to have you as the user focus on the query and business case. The work associated with managing the hardware needed to run your Stream Analytics job is abstracted – leaving you worry-free.
Like many other pay-as-you-go Azure offerings, ASA is priced by the number of Streaming Units provisioned.
Now, with all of this out of the way, let’s dive into how you determine how many SUs you need to run your job successfully.
ASA jobs perform all processing in memory, as such, when a job runs out of memory, the streaming job fails. Given this, you need to make sure you have enough SUs for your job to run continuously, 24/7.
How do you ensure you have enough SUs for your job to run continuously? You set up alerts and continuously monitor your job’s resource usage to ensure enough resources are allocated to keep the job running. One metric to keep a close eye on is the SU % Utilization metric. This metric ranges from 0% to 100%and describes the memory consumption of a job’s workload. ASA users are advised to set alerts on this metric and once triggered, they have an opportunity to intervene and scale the job’s streaming units appropriately. This means, at any point in time, when an alert is triggered, human intervention might be needed.
Until now…. Enter Autoscale.
Autoscale allows you to have the right number of resources running to handle the load on your job. It will add resources to handle increases in load and save money by removing resources that are sitting idle. How Autoscale behaves is determined by you -- you set the minimum, maximum, and configure the scaling behavior using custom rules.
How exactly does that work? Let’s look at some examples.
We talked about the SU % Utilization metric and the need to continuously monitor that metric.
With Autoscale, you can set the following rules with only a few clicks:
- Increase number of SUs allocated to my job when the average SU % Utilization metric goes above 75% for 10 minutes.
- Decrease number of SUs allocated to my job when the average SU % Utilization metric goes below 30% for 15 minutes.
Or simply,
- Use more Streaming Units during business hours and fewer during off hours.
There are many variations of these rules, but the most important thing is that you set the rules and walk away. Autoscale will keep your job running while abiding by your rules.
Do I have to write code to enable Autoscale rules?
Nope.
Configuring Autoscale settings is straightforward. Simply log onto Azure portal, find your Stream Analytics job, and look for Scale on the menu on the left. At this point, you have already completed 1/3 of the work.
Once you click on Scale, choose Custom Autoscale and Voilà! From here on, you can begin playing with the different conditions, scale modes, and rules.
To read the full How to Guide, visit Azure Stream Analytics autoscale streaming units | Microsoft Docs.
Any examples of ASA customer scenarios?
Yes. One ASA customer uses Autoscale to scale up or down as needed to successfully process the data associated with insect movements and rain drops for their real-time field sensors.
Without previous trends and patterns, this customer had a hard time anticipating the correct volume of insect movements, their problem was further exacerbated during rainy seasons where raindrops would set off the sensors. Autoscale optimized this customer's job performance and resolved any issues with overprovisioning SUs.
To summarize…
Whether you have one job or various jobs, scaling your jobs manually can create a lot of operational overhead. Autoscale is your solution to running your job optimally while keeping costs low.
The best of both worlds—increased throughput and reduced costs—is now available with ASA’s Autoscale.