Azure Data Explorer Quarterly Newsletter (September 2022)
Published Sep 15 2022 05:43 AM 1,392 Views
Microsoft

Here is a recap on the latest and greatest features and updates introduced in the last few months. In case you have missed them they are broken down into the following sections:

 

Cost and Performance Improvements

Tools and User Experience

Data Ingestion

Engine

Learning and Readiness

 

Cost and Performance Improvements

Free Cluster Upgrade Option – Cost and Performance Improvements

An in-place seamless upgrade from the free cluster to a full-blown ADX cluster that offers additional benefits. The upgrade path is easy and can be done with a few clicks through the wizard which allows you to associate your cluster with an Azure subscription.

MyFreeClusters.png

Optimized Auto Scale – Predictive – Cost and Performance Improvements

The  optimized autoscale feature has been in place and using reactive logic. It has been helping ADX users by adjusting the cluster size when there is an increase in resources due to ingestion or query load.  The optimized autoscale feature has been further improved with predictive logic. This logic monitors the same metrics as the reactive logic and over time builds up the cluster usage pattern and uses this to forecast and plan the size of the cluster.  The reactive logic is still used to ensure any forecast anomalies or usage pattern changes are still autoscaled appropriately.

Autoscale.png

 

SKU migration improvements

Kusto has released a major improvement to the Scaleup/SKU migration process. Till now, when a cluster performed a SKU migration, the cluster would stop providing service while the new VMSS of the new SKU was up and running. In the new mechanism we create the cluster's VMSS with the new SKU in parallel to the existing VMSS and then switchover when it is ready to take over.  This optimization reduces the downtime experienced during SKU migrations, in many cases makes the migration a seamless process. 

 

ADX Benchmark Report

Microsoft invited GigaOm to measure the performance of the Azure Data Explorer engine and compare it with its leading competitors, BigQuery and Snowflake..

 

The results, now available in the GigaOm report, show that Azure Data Explorer provides superior performance at a significantly lower cost in both single and high-concurrency scenarios.

For example, the following chart taken from the report displays the results of executing the benchmark while simulating 50 concurrent users:

Gigaom.png

For further insights, we highly recommend reading the full Log and Telemetry Analytics Performance Benchmark report. And don’t just take our word for it.

 

Azure Data Explorer: Log and telemetry analytics benchmark | Azure Blog and Updates | Microsoft Azur...

 

Tools and User Experience

 ADX Dashboard Improvements

A number of improvements have been made to dashboards. You now have the ability to export and import dashboards from a file (json format).  This allows users to add version control, for scenarios where manual editing is preferred or if you want to create a baseline to be used as a template to share with other teams.

 

The cross filter feature allows you to select a value in one visual which is reflected in all visuals on the dashboard to show the related date. To learn more about setting up cross filters take a look at the dashboard parameters guide.

 

The drill through feature allows you to select a value in a visual to show further information related to the value selected. The drill through is shown in a target page on the same dashboard. To learn more about setting up drill throughs follow this guide.

ShafMahmood_3-1662729944043.png

 

 

Grafana – Query Builder

The Grafana plug-in allows you to connect to and visualize data from Azure Data Explorer. After adding ADX as a data source you can make use of the query editor to start building your queries using the query builder mode.

 

Grafana.png

 

Web Explorer Sample Gallery

A great way to learn about a product is to see how it is being used by others. The Web Explorer sample gallery provides end-to-end samples of how customers leverage Azure Data Explorer popular use cases such as Logs Data, Metrics Data, IoT data and Basic big data examples. Each sample includes the dataset, well-documented queries, and a sample dashboard.   

 

Navigate to the sample gallery from the Web Explorer homepage. From the homepage’s Get Started section, go to Explore sample data with KQL to access the sample queries or go to Explore sample dashboards to open the sample dashboards.   

 

ExploreDataSamples.png

 

To access the sample gallery, you need either an Azure active directory (AAD) user identity or a Microsoft account (MSA)

 

To learn more about the sample gallery, read Azure Data Explorer in 60 minutes with the new samples' gallery  

 

Efficient joins for relationships in Power BI

Starting from the September version of PBI desktop, the Kusto connector was updated. You can now add to the source statement in Power Query for dimension tables the clause: IsDimension=true

 

ShafMahmood_6-1662729944054.png

You can use both dimensions and fact tables in Direct Query mode and create relationships between them after you add this setting.

For further information please refer to Return of the join in Power BI with Kusto.

 

Kusto Emulator

We are excited to announce the General Availability (GA) of the Kusto Emulator.

 

ShafMahmood_7-1662729944059.png

 

The Kusto Emulator is a Docker Image exposing a Kusto Query Engine endpoint.  We can use it to create databases, ingest and query data.  The emulator understands Kusto Query Language (KQL) the same way the Azure Data Explorer Service does.

 

ADX Web Explorer August Updates

Go and check out the August ADX Web Explorer update blog which highlights a number of improvements and features listed below that have been recently introduced:

  • Dashboard iframe embedding
  • Funnel visual
  • Status bar now supporting new stats for Datetime datatype
  • Close a single tab functionality
  • .Net and Node.jS support in Sample App Generator
  • Simplifying JSON selection

 

New Productivity features in Kusto Explorer (Desktop version)

New features have been in introduced Kusto Explorer (Desktop version of Azure Data Explorer Web UI) to help you be more productive in managing and executing queries.

 

The introduction of Unsaved Work and Traced Folders help you organize your queries and Query Automation allows you to define a workflow that contains a series of queries with rules and logic that govern the order in which they are executed.

 

Take a look at this blog which has more details.

 

Native Parquet Export

This new feature allows a more efficient export into Parquet. Its faster and creates smaller output blobs that are more efficient to query.

 

To make use of this feature set the useNativeParquetWriter to true (default is false) in the one-time Export command or when creating Continuous Data Export.

 

Example: .export to table externalTableParquet with (useNativeParquetWriter = true) <| Usage

 

bag_has_key() Function

The new function is now available and allows you to check if a given dynamic bag column has a key.

For more information and example syntax please refer to the documentation.

Data Ingestion

Serilog Sink

A Serilog sink that writes events to an Azure Data Explorer (Kusto) cluster.  This initial release has features to support both queued and streaming ingestion, data mappings, AAD user and application authentication for ADX, Azure Synapse Data Explorer and the ADX Free Cluster.

ShafMahmood_8-1662729944061.png

 

Ingest from Amazon S3

We are excited to launch the ability to ingest data from Amazon Simple Storage Service (S3)  into Azure Data Explorer (ADX) natively. With the new S3 support, customers can bring data from S3 natively without relying on complex ETL pipelines.

The .ingest into ADX command ingests data into a table by "pulling" the data from one or more cloud storage files.  The command now supports Amazon S3 URLs with below syntax. Read more in the docs.

Ingest a file using IAM credentials:

ingest.png

For more information and to understand the best use of this option please check out this blog.

 

Azure Synapse Data Explorer connector for Microsoft Power Automate, Logic Apps, and Power Apps

The Azure Data Explorer connector for Power Automate  is now GA with many features that allow you to orchestrate and schedule flows, send notifications, and alerts, as part of a scheduled or triggered task. 

 

In Power Automate, you can: 

  • Send notifications and alerts based on query results, such as when thresholds exceed certain limits. 
  • Send regular, such as daily or weekly, reports containing tables and charts. 
  • Schedule regular jobs using control commands on clusters. For example, copy data from one table to another using the  .set-or-append command. 
  • Export and import data between Azure Data Explorer and other databases. 

ShafMahmood_10-1662729944070.png

Engine

Conversion functions

Are you working with measurement data and need to convert from one unit to another, the new conversation functions are a great way to do this.

 

Conversation functions have been added for the following measurements: Angle, Energy, Force, Length, Mass, Speed, Temperature and Volume.

 

Below is an example to convert from Kelvin to DegreeCelsius:

 

ShafMahmood_11-1662729944071.png

 

Functions library

The Functions Library has several useful user-defined functions for Machine Learning, PromQL, Series Processing Functions and Statistical and Probably functions.

 

The documentation has been updated to provide further details.

 

Output Schema

The getschema operator has been updated to allow you to specify the output to be shown in an easy and reusable format using the parameter kind=csl.

 

ShafMahmood_12-1662729944074.png

You can use the output as syntax for plugins that require it.

For more information refer to this blog.

Learning and Readiness

Kusto Detective Agency is looking for new recruits

Do you want to be a certified as a Kusto Detective? To become a detective, you need to complete some Kusto assignments. Those who complete the journey will become full-fledged detectives and be awarded special badges!

Join Now

 

KustoDetective.png

 

 

ADX Kusto Microhack

Kusto Product Group and Microsoft Global Black Belt team are pleased to present this challenge based, collaboration driven, discover-by-doing learning experience to you. Microhacks are divided into three parts to cater enough time for the participants to understand the key concepts of Azure Data Explorer effectively.

 

  • Microhack 1: Cluster creation and data ingestion. This Microhack will focus on enabling the participants to design ADX based big data analytics solution, create an ADX cluster, and ingest data into the cluster.
  • Microhack 2: Data exploration and visualization using Kusto Query Language (KQL). This Microhack will focus on enabling the participants to write Kusto queries to explore and analyze the data stored in their clusters. Participants will also create cool visualizations. It is recommended to complete the Microhack 1 before beginning with Microhack 2.
  • Microhack 3: Advanced capabilities.  This Microhack will focus on enabling the participants to create Materialized Views, Functions, and use advanced operators to explore and analyze the data.

ShafMahmood_13-1662730011529.png

 

Participants also have the opportunity to earn an ADX MicroHack Digital Badge!

 

Kusto Learning On YouTube

There is some excellent readiness content that you can consume via the ADX YouTube channel!

 

The series will be updated to consist of a number of short episodes designed to help you gain an understanding of different capabilities within Azure Data Explorer, also known as ADX or Kusto.

 

In the first video of the series, Tzvia Gitlin Troyna, Principal Program Manager at the Azure Data Explorer team, will give us a quick introduction on ADX, what it is, when to use it, and how to get started with ADX.

 

The following sessions are currently in the pipeline to be released soon:

 

  • Data Ingestion
  • Sample data flow

The channel has the ADX L300 workshop series. If you have not already seen this, head over and check them out.

 

Please subscribe and share the channel details.

Azure Data Explorer - YouTube

 

ADX 101 learning series on YouTube 

In addition, for those who are new to Kusto, we have started a 101 learning video series.

Check out the 4 recently released videos of the Azure Data Explorer 101 series

 

 

 

1 Comment
Version history
Last update:
‎Sep 18 2022 11:05 PM
Updated by: