Here is a recap on the latest and greatest features and updates introduced in the last few months. In case you have missed them they are broken down into the following sections:
Free Cluster Upgrade Option – Cost and Performance Improvements
An in-place seamless upgrade from the free cluster to a full-blown ADX cluster that offers additional benefits. The upgrade path is easy and can be done with a few clicks through the wizard which allows you to associate your cluster with an Azure subscription.
Optimized Auto Scale – Predictive – Cost and Performance Improvements
The optimized autoscale feature has been in place and using reactive logic. It has been helping ADX users by adjusting the cluster size when there is an increase in resources due to ingestion or query load. The optimized autoscale feature has been further improved with predictive logic. This logic monitors the same metrics as the reactive logic and over time builds up the cluster usage pattern and uses this to forecast and plan the size of the cluster. The reactive logic is still used to ensure any forecast anomalies or usage pattern changes are still autoscaled appropriately.
SKU migration improvements
Kusto has released a major improvement to the Scaleup/SKU migration process. Till now, when a cluster performed a SKU migration, the cluster would stop providing service while the new VMSS of the new SKU was up and running. In the new mechanism we create the cluster's VMSS with the new SKU in parallel to the existing VMSS and then switchover when it is ready to take over. This optimization reduces the downtime experienced during SKU migrations, in many cases makes the migration a seamless process.
ADX Benchmark Report
Microsoft invited GigaOm to measure the performance of the Azure Data Explorer engine and compare it with its leading competitors, BigQuery and Snowflake..
The results, now available in the GigaOm report, show that Azure Data Explorer provides superior performance at a significantly lower cost in both single and high-concurrency scenarios.
For example, the following chart taken from the report displays the results of executing the benchmark while simulating 50 concurrent users:
For further insights, we highly recommend reading the full Log and Telemetry Analytics Performance Benchmark report. And don’t just take our word for it.
ADX Dashboard Improvements
A number of improvements have been made to dashboards. You now have the ability to export and import dashboards from a file (json format). This allows users to add version control, for scenarios where manual editing is preferred or if you want to create a baseline to be used as a template to share with other teams.
The cross filter feature allows you to select a value in one visual which is reflected in all visuals on the dashboard to show the related date. To learn more about setting up cross filters take a look at the dashboard parameters guide.
The drill through feature allows you to select a value in a visual to show further information related to the value selected. The drill through is shown in a target page on the same dashboard. To learn more about setting up drill throughs follow this guide.
Grafana – Query Builder
The Grafana plug-in allows you to connect to and visualize data from Azure Data Explorer. After adding ADX as a data source you can make use of the query editor to start building your queries using the query builder mode.
Web Explorer Sample Gallery
A great way to learn about a product is to see how it is being used by others. The Web Explorer sample gallery provides end-to-end samples of how customers leverage Azure Data Explorer popular use cases such as Logs Data, Metrics Data, IoT data and Basic big data examples. Each sample includes the dataset, well-documented queries, and a sample dashboard.
Navigate to the sample gallery from the Web Explorer homepage. From the homepage’s Get Started section, go to Explore sample data with KQL to access the sample queries or go to Explore sample dashboards to open the sample dashboards.
To access the sample gallery, you need either an Azure active directory (AAD) user identity or a Microsoft account (MSA).
To learn more about the sample gallery, read Azure Data Explorer in 60 minutes with the new samples' gallery
Efficient joins for relationships in Power BI
Starting from the September version of PBI desktop, the Kusto connector was updated. You can now add to the source statement in Power Query for dimension tables the clause: IsDimension=true
You can use both dimensions and fact tables in Direct Query mode and create relationships between them after you add this setting.
For further information please refer to Return of the join in Power BI with Kusto.
We are excited to announce the General Availability (GA) of the Kusto Emulator.
The Kusto Emulator is a Docker Image exposing a Kusto Query Engine endpoint. We can use it to create databases, ingest and query data. The emulator understands Kusto Query Language (KQL) the same way the Azure Data Explorer Service does.
ADX Web Explorer August Updates
Go and check out the August ADX Web Explorer update blog which highlights a number of improvements and features listed below that have been recently introduced:
New Productivity features in Kusto Explorer (Desktop version)
New features have been in introduced Kusto Explorer (Desktop version of Azure Data Explorer Web UI) to help you be more productive in managing and executing queries.
The introduction of Unsaved Work and Traced Folders help you organize your queries and Query Automation allows you to define a workflow that contains a series of queries with rules and logic that govern the order in which they are executed.
Take a look at this blog which has more details.
Native Parquet Export
This new feature allows a more efficient export into Parquet. Its faster and creates smaller output blobs that are more efficient to query.
Example: .export to table externalTableParquet with (useNativeParquetWriter = true) <| Usage
The new function is now available and allows you to check if a given dynamic bag column has a key.
For more information and example syntax please refer to the documentation.
A Serilog sink that writes events to an Azure Data Explorer (Kusto) cluster. This initial release has features to support both queued and streaming ingestion, data mappings, AAD user and application authentication for ADX, Azure Synapse Data Explorer and the ADX Free Cluster.
Ingest from Amazon S3
We are excited to launch the ability to ingest data from Amazon Simple Storage Service (S3) into Azure Data Explorer (ADX) natively. With the new S3 support, customers can bring data from S3 natively without relying on complex ETL pipelines.
The .ingest into ADX command ingests data into a table by "pulling" the data from one or more cloud storage files. The command now supports Amazon S3 URLs with below syntax. Read more in the docs.
Ingest a file using IAM credentials:
For more information and to understand the best use of this option please check out this blog.
Azure Synapse Data Explorer connector for Microsoft Power Automate, Logic Apps, and Power Apps
The Azure Data Explorer connector for Power Automate is now GA with many features that allow you to orchestrate and schedule flows, send notifications, and alerts, as part of a scheduled or triggered task.
In Power Automate, you can:
Are you working with measurement data and need to convert from one unit to another, the new conversation functions are a great way to do this.
Conversation functions have been added for the following measurements: Angle, Energy, Force, Length, Mass, Speed, Temperature and Volume.
Below is an example to convert from Kelvin to DegreeCelsius:
The Functions Library has several useful user-defined functions for Machine Learning, PromQL, Series Processing Functions and Statistical and Probably functions.
The documentation has been updated to provide further details.
The getschema operator has been updated to allow you to specify the output to be shown in an easy and reusable format using the parameter kind=csl.
You can use the output as syntax for plugins that require it.
For more information refer to this blog.
Kusto Detective Agency is looking for new recruits
Do you want to be a certified as a Kusto Detective? To become a detective, you need to complete some Kusto assignments. Those who complete the journey will become full-fledged detectives and be awarded special badges!
ADX Kusto Microhack
Kusto Product Group and Microsoft Global Black Belt team are pleased to present this challenge based, collaboration driven, discover-by-doing learning experience to you. Microhacks are divided into three parts to cater enough time for the participants to understand the key concepts of Azure Data Explorer effectively.
Participants also have the opportunity to earn an ADX MicroHack Digital Badge!
Kusto Learning On YouTube
There is some excellent readiness content that you can consume via the ADX YouTube channel!
The series will be updated to consist of a number of short episodes designed to help you gain an understanding of different capabilities within Azure Data Explorer, also known as ADX or Kusto.
In the first video of the series, Tzvia Gitlin Troyna, Principal Program Manager at the Azure Data Explorer team, will give us a quick introduction on ADX, what it is, when to use it, and how to get started with ADX.
The following sessions are currently in the pipeline to be released soon:
The channel has the ADX L300 workshop series. If you have not already seen this, head over and check them out.
Please subscribe and share the channel details.
ADX 101 learning series on YouTube
In addition, for those who are new to Kusto, we have started a 101 learning video series.
Check out the 4 recently released videos of the Azure Data Explorer 101 series.
You must be a registered user to add a comment. If you've already registered, sign in. Otherwise, register and sign in.