Have you ever faced a situation whereyour streaming jobproduces no result or unexpected results and you don’t know how to trouble shoot which parts go wrong?We are happy to announce the roll out of the newest debugging feature– job diagram debugging inVisual Studio Code extension for Azure Stream Analytics.This feature brings together job diagrams, metrics, diagnostic logs, and intermediate results to help you quickly isolate the source of a problem. You can not only test your queryon your local machinebut also connect to live sources such as Event Hub, IoT Hub.Thejob diagram helps you to understand how data flows between each step, and you can view the intermediate result set and metricsto debug issues.You can iterate fast because eachlocal testof the job only takes seconds.
This blog will use a real example to show you how to debug an Azure Stream Analytics job using job diagramin Visual Studio Code.
This job diagram only shows the data and metrics for local testing in a single node on your own machine. It should not be used for performance and scalability tuning.
Sign-in Visual Studio Code using your Azure account.
Step 2 Open your job in Visual Studio Code
Please go to step 3 if you already have your Azure Stream Analytics project opened in Visual Studio Code.
Open the Azure Stream Analytics job you want to debug onthe Query Editor on Azure portal. Select from drop down Open in Visual Studio and choose Visual Studio Code. Then choose Open job in Visual Studio Code.The job will be exported as an Azure Stream Analytics project in Visual Studio Code.
Step 3 Run job locally
Since the credentials are already auto-populated, only thing you need to do is to open the script and Run locally. Make sure the input data is sending to your job input sources.
Step 4 Debug using job diagram
The job diagramshown on the right window of the editor showshow data flows from input sources, like Event Hub or IoT Hub, through multiple query steps to output sinks.You can also view the data as well as metrics of each query step in each intermediate result set to find the source of an issue.
Now, let’s look at a real example job below. We have a job receiving stock quotes for different stocks. In the query there are filters on a few stocks but one output does not have any data.
We run the joblocally against live input stream from Event Hub. Through job diagram we can see that the step ‘msftquotes’ does not have data flowing in.
To troubleshoot that, let’s zoom in the diagram and select the upstream step ‘typeconvertedquotes’ to see if there is any output.In the node it shows that there are 3135 output events. Also, from the Result tab below it is easy to find out that there are data been output from this step with symbol ‘MSFT’.
Then we select the step ‘msftquotes’ and locate to the corresponding script to take a closer look.
Now we find the root cause – there is a typo in the script, ‘%MSFT%’ is mistakenly typed as ‘%MSFA%’.