Azure Data Explorer services for storing and running interactive analytics Big Data

Published 07-05-2019 04:14 AM 2,356 Views
Microsoft

So this week I have had a number of questions around course development of Big Data Analysis technologies and number of queries have been specifically around Azure Data Explorer (Kusto), the following blog is a quick overview of Kusto / Azure Data Explorer.

Azure Data Explorer is a big data analytics cloud service optimized for interactive ad-hoc queries over structured, semi-structured, and unstructured data. Kusto is the internal code name of the project in Microsoft. Externally, the cloud service is called Azure Data Explorer.

 

Kusto is a log analytics cloud platform optimized for ad-hoc big data queries. You can read more about Kusto here: https://docs.microsoft.com/en-us/azure/kusto/

 

The world of Big Data is growing steadily, and the number of technologies that process large amounts of data is growing along with it. So how does Kusto compare to other tools such as Cosmos, MDM and Hadoop. So firstly lets consider the three telemetry-processing scenarios, based primarily on latency needs:

  1. Hot path
  2. Warm path
  3. Cold path

For example MDM,  traditional TSDBs, and many stream processing technologies such as Azure Stream Analytics are considered as "hot path" technologies.

Kusto targets the "warm path" scenario


Various batch processing systems (such as Cosmos, Hadoop, and Azure Data Lake Compute) are "cold path".

The following table attempts to highlight some of the differences.

Aspect Hot path Warm path Cold path
Latency Seconds (up to, say, 5) Minutes (up to, say, five) More
Queryable data storage RAM Attached (low latency) SSD HDD (Cosmos, Hadoop) or even remote storage (HDInsight)
Query frequency Automated (alerting) Ad-hoc (human-generated) Occasional
Max size of intermediate result Single-node RAM Cluster RAM "Infinite" (spilled to HDD)
Recovery from query failures No No Yes (built for batch processing; continue from last checkpoint)
Data analysis Metrics (TSDB-like) Text and numbers Everything you can write a C# function for
Data form Aggregated Raw Raw
Targeted for Real time data viewing Ad-hoc data exploration Programmatic data manipulation

Kusto is built for analytics, rather than OLTP, scenarios. Therefore, it design trade-offs favor very fast bulk Create (supporting high rates of inserts/appends of new records) and very fast bulk Read (supporting queries over large amounts of data). Kusto's support for Delete scenarios focuses on bulk-delete (mainly for retention period), and per-record deletion is not supported. Likewise, Updates of existing records is not supported in Kusto.

Kusto offers excellent data ingestion and query performance by "sacrificing" the ability to perform in-place updates of individual rows and cross-table constraints/transactions. Therefore, it supplants, rather than replaces, traditional RDBMS systems for scenarios such as OLTP and data warehousing.

 

As a Big Data service, Kusto handles structured, semi-structured e.g. JSON-like nested type

 

Introductory videos

Azure Data Explorer was first announced in Ignite 2018

  1. Scott Guthrie's announcement in Orlando: https://www.youtube.com/watch?v=xnmBu4oh7xk&t=1h08m12s
  2. Rohan Kumar's announcement: https://www.youtube.com/watch?v=ZaiM89Z01r0&t=58m0s ]
  3. Manoj Raheja's brief introduction to Kusto: https://www.youtube.com/watch?v=GT4C84yrb68
  4. Scott Guthrie demoing Kusto in Techorama: 
    https://www.youtube.com/watch?v=YTWewM_UMOk&feature=youtu.be&t=3074

Kusto is used as the data platform for a number of Microsoft services, some of which expose its query language to users. Here are two videos showing its capabilities when used inside Application Insights / Azure Monitor:

  1. Interactive Analytics with Application Insights
  2. Advanced Analytics with Application Insights

Product links:

Social:

%3CLINGO-SUB%20id%3D%22lingo-sub-739121%22%20slang%3D%22en-US%22%3EAzure%20Data%20Explorer%20services%20for%20storing%20and%20running%20interactive%20analytics%20Big%20Data%3C%2FLINGO-SUB%3E%3CLINGO-BODY%20id%3D%22lingo-body-739121%22%20slang%3D%22en-US%22%3E%3CP%3ESo%20this%20week%20I%20have%20had%20a%20number%20of%20questions%20around%20course%20development%20of%20Big%20Data%20Analysis%20technologies%20and%20number%20of%20queries%20have%20been%20specifically%20around%20Azure%20Data%20Explorer%20(Kusto)%2C%20the%20following%20blog%20is%20a%20quick%20overview%20of%26nbsp%3BKusto%26nbsp%3B%2F%26nbsp%3BAzure%20Data%20Explorer.%3CBR%20%2F%3E%3CBR%20%2F%3E%3CSPAN%3EAzure%20Data%20Explorer%20is%20a%20big%20data%20analytics%20cloud%20service%20optimized%20for%20interactive%20ad-hoc%20queries%20over%20structured%2C%20semi-structured%2C%20and%20unstructured%20data.%20Kusto%20is%20the%20internal%20code%20name%20of%20the%20project%20in%20Microsoft.%20Externally%2C%20the%20cloud%20service%20is%20called%26nbsp%3B%3C%2FSPAN%3E%3CA%20href%3D%22https%3A%2F%2Fazure.microsoft.com%2Fen-us%2Fservices%2Fdata-explorer%2F%22%20target%3D%22_blank%22%20rel%3D%22noopener%20noreferrer%22%3EAzure%20Data%20Explorer%3C%2FA%3E%3CSPAN%3E.%3C%2FSPAN%3E%3C%2FP%3E%0A%3CP%3E%26nbsp%3B%3C%2FP%3E%0A%3CP%3E%3CIMG%20src%3D%22https%3A%2F%2Fconnectoricons-prod.azureedge.net%2Fkusto%2Ficon_1.0.1152.1466.png%22%20border%3D%220%22%20%2F%3E%3CSPAN%3EKusto%20is%20a%20log%20analytics%20cloud%20platform%20optimized%20for%20ad-hoc%20big%20data%20queries.%20You%20can%20read%20more%20about%20Kusto%20here%3A%26nbsp%3B%3C%2FSPAN%3E%3CA%20href%3D%22https%3A%2F%2Fdocs.microsoft.com%2Fen-us%2Fazure%2Fkusto%2F%22%20target%3D%22_blank%22%20rel%3D%22noopener%20noreferrer%22%3Ehttps%3A%2F%2Fdocs.microsoft.com%2Fen-us%2Fazure%2Fkusto%2F%3C%2FA%3E%3C%2FP%3E%0A%3CP%3E%26nbsp%3B%3C%2FP%3E%0A%3CP%3EThe%20world%20of%20Big%20Data%20is%20growing%20steadily%2C%20and%20the%20number%20of%20technologies%20that%20process%20large%20amounts%20of%20data%20is%20growing%20along%20with%20it.%20So%20how%20does%20Kusto%20compare%20to%20other%20tools%20such%20as%20Cosmos%2C%20MDM%20and%20Hadoop.%20So%20firstly%20lets%20consider%20the%20three%20telemetry-processing%20scenarios%2C%20based%20primarily%20on%20latency%20needs%3A%3CBR%20%2F%3E%3CBR%20%2F%3E%3C%2FP%3E%0A%3COL%3E%0A%3CLI%3EHot%20path%3C%2FLI%3E%0A%3CLI%3EWarm%20path%3C%2FLI%3E%0A%3CLI%3ECold%20path%3C%2FLI%3E%0A%3C%2FOL%3E%0A%3CP%3EFor%20example%20MDM%2C%26nbsp%3B%20traditional%20TSDBs%2C%20and%20many%20stream%20processing%20technologies%20such%20as%20Azure%20Stream%20Analytics%20are%20considered%20as%20%22hot%20path%22%20technologies.%20%3CBR%20%2F%3E%3CBR%20%2F%3EKusto%20targets%20the%20%22warm%20path%22%20scenario%3C%2FP%3E%0A%3CP%3E%3CBR%20%2F%3EVarious%20batch%20processing%20systems%20(such%20as%20Cosmos%2C%20Hadoop%2C%20and%20Azure%20Data%20Lake%20Compute)%20are%20%22cold%20path%22.%3CBR%20%2F%3E%3CBR%20%2F%3E%3C%2FP%3E%0A%3CP%3EThe%20following%20table%20attempts%20to%20highlight%20some%20of%20the%20differences.%3CBR%20%2F%3E%3CBR%20%2F%3E%3C%2FP%3E%0A%3CDIV%20class%3D%22table-responsive%22%3E%0A%3CTABLE%20class%3D%22table%20table-bordered%20table-striped%20table-condensed%22%3E%0A%3CTHEAD%3E%0A%3CTR%3E%0A%3CTH%3EAspect%3C%2FTH%3E%0A%3CTH%3EHot%20path%3C%2FTH%3E%0A%3CTH%3EWarm%20path%3C%2FTH%3E%0A%3CTH%3ECold%20path%3C%2FTH%3E%0A%3C%2FTR%3E%0A%3C%2FTHEAD%3E%0A%3CTBODY%3E%0A%3CTR%3E%0A%3CTD%3ELatency%3C%2FTD%3E%0A%3CTD%3ESeconds%20(up%20to%2C%20say%2C%205)%3C%2FTD%3E%0A%3CTD%3EMinutes%20(up%20to%2C%20say%2C%20five)%3C%2FTD%3E%0A%3CTD%3EMore%3C%2FTD%3E%0A%3C%2FTR%3E%0A%3CTR%3E%0A%3CTD%3EQueryable%20data%20storage%3C%2FTD%3E%0A%3CTD%3ERAM%3C%2FTD%3E%0A%3CTD%3EAttached%20(low%20latency)%20SSD%3C%2FTD%3E%0A%3CTD%3EHDD%20(Cosmos%2C%20Hadoop)%20or%20even%20remote%20storage%20(HDInsight)%3C%2FTD%3E%0A%3C%2FTR%3E%0A%3CTR%3E%0A%3CTD%3EQuery%20frequency%3C%2FTD%3E%0A%3CTD%3EAutomated%20(alerting)%3C%2FTD%3E%0A%3CTD%3EAd-hoc%20(human-generated)%3C%2FTD%3E%0A%3CTD%3EOccasional%3C%2FTD%3E%0A%3C%2FTR%3E%0A%3CTR%3E%0A%3CTD%3EMax%20size%20of%20intermediate%20result%3C%2FTD%3E%0A%3CTD%3ESingle-node%20RAM%3C%2FTD%3E%0A%3CTD%3ECluster%20RAM%3C%2FTD%3E%0A%3CTD%3E%22Infinite%22%20(spilled%20to%20HDD)%3C%2FTD%3E%0A%3C%2FTR%3E%0A%3CTR%3E%0A%3CTD%3ERecovery%20from%20query%20failures%3C%2FTD%3E%0A%3CTD%3ENo%3C%2FTD%3E%0A%3CTD%3ENo%3C%2FTD%3E%0A%3CTD%3EYes%20(built%20for%20batch%20processing%3B%20continue%20from%20last%20checkpoint)%3C%2FTD%3E%0A%3C%2FTR%3E%0A%3CTR%3E%0A%3CTD%3EData%20analysis%3C%2FTD%3E%0A%3CTD%3EMetrics%20(TSDB-like)%3C%2FTD%3E%0A%3CTD%3EText%20and%20numbers%3C%2FTD%3E%0A%3CTD%3EEverything%20you%20can%20write%20a%20C%23%20function%20for%3C%2FTD%3E%0A%3C%2FTR%3E%0A%3CTR%3E%0A%3CTD%3EData%20form%3C%2FTD%3E%0A%3CTD%3EAggregated%3C%2FTD%3E%0A%3CTD%3ERaw%3C%2FTD%3E%0A%3CTD%3ERaw%3C%2FTD%3E%0A%3C%2FTR%3E%0A%3CTR%3E%0A%3CTD%3ETargeted%20for%3C%2FTD%3E%0A%3CTD%3EReal%20time%20data%20viewing%3C%2FTD%3E%0A%3CTD%3EAd-hoc%20data%20exploration%3C%2FTD%3E%0A%3CTD%3EProgrammatic%20data%20manipulation%3C%2FTD%3E%0A%3C%2FTR%3E%0A%3C%2FTBODY%3E%0A%3C%2FTABLE%3E%0A%3C%2FDIV%3E%0A%3CP%3EKusto%20is%20built%20for%20analytics%2C%20rather%20than%20OLTP%2C%20scenarios.%20Therefore%2C%20it%20design%20trade-offs%20favor%20very%20fast%20bulk%3CSPAN%3E%26nbsp%3B%3C%2FSPAN%3E%3CSTRONG%3EC%3C%2FSTRONG%3Ereate%20(supporting%20high%20rates%20of%20inserts%2Fappends%20of%20new%20records)%20and%20very%20fast%20bulk%3CSPAN%3E%26nbsp%3B%3C%2FSPAN%3E%3CSTRONG%3ER%3C%2FSTRONG%3Eead%20(supporting%20queries%20over%20large%20amounts%20of%20data).%20Kusto's%20support%20for%3CSPAN%3E%26nbsp%3B%3C%2FSPAN%3E%3CSTRONG%3ED%3C%2FSTRONG%3Eelete%20scenarios%20focuses%20on%20bulk-delete%20(mainly%20for%20retention%20period)%2C%20and%20per-record%20deletion%20is%20not%20supported.%20Likewise%2C%3CSPAN%3E%26nbsp%3B%3C%2FSPAN%3E%3CSTRONG%3EU%3C%2FSTRONG%3Epdates%20of%20existing%20records%20is%20not%20supported%20in%20Kusto.%3CBR%20%2F%3E%3CBR%20%2F%3E%3C%2FP%3E%0A%3CP%3EKusto%20offers%20excellent%20data%20ingestion%20and%20query%20performance%20by%20%22sacrificing%22%20the%20ability%20to%20perform%20in-place%20updates%20of%20individual%20rows%20and%20cross-table%20constraints%2Ftransactions.%20Therefore%2C%20it%20supplants%2C%20rather%20than%20replaces%2C%20traditional%20RDBMS%20systems%20for%20scenarios%20such%20as%20OLTP%20and%20data%20warehousing.%3C%2FP%3E%0A%3CP%3E%26nbsp%3B%3C%2FP%3E%0A%3CP%3EAs%20a%20Big%20Data%20service%2C%20Kusto%20handles%20structured%2C%20semi-structured%20e.g.%20JSON-like%20nested%20type%3C%2FP%3E%0A%3CH2%20id%3D%22toc-hId-1790766613%22%20id%3D%22toc-hId-1790766611%22%3E%26nbsp%3B%3C%2FH2%3E%0A%3CH2%20id%3D%22introductory-videos-public%22%20id%3D%22toc-hId--761390348%22%20id%3D%22toc-hId--761390350%22%3EIntroductory%20videos%3C%2FH2%3E%0A%3CP%3EAzure%20Data%20Explorer%20was%20first%20announced%20in%20Ignite%202018%3C%2FP%3E%0A%3COL%3E%0A%3CLI%3EScott%20Guthrie's%20announcement%20in%20Orlando%3A%3CSPAN%3E%26nbsp%3B%3C%2FSPAN%3E%3CA%20href%3D%22https%3A%2F%2Fwww.youtube.com%2Fwatch%3Fv%3DxnmBu4oh7xk%26amp%3Bt%3D1h08m12s%22%20target%3D%22_blank%22%20rel%3D%22noopener%20nofollow%20noreferrer%22%3Ehttps%3A%2F%2Fwww.youtube.com%2Fwatch%3Fv%3DxnmBu4oh7xk%26amp%3Bt%3D1h08m12s%3C%2FA%3E%3C%2FLI%3E%0A%3CLI%3ERohan%20Kumar's%20announcement%3A%3CSPAN%3E%26nbsp%3B%3C%2FSPAN%3E%3CA%20href%3D%22https%3A%2F%2Fwww.youtube.com%2Fwatch%3Fv%3DZaiM89Z01r0%26amp%3Bt%3D58m0s%22%20target%3D%22_blank%22%20rel%3D%22noopener%20nofollow%20noreferrer%22%3Ehttps%3A%2F%2Fwww.youtube.com%2Fwatch%3Fv%3DZaiM89Z01r0%26amp%3Bt%3D58m0s%20%5D%3C%2FA%3E%3C%2FLI%3E%0A%3CLI%3EManoj%20Raheja's%20brief%20introduction%20to%20Kusto%3A%3CSPAN%3E%26nbsp%3B%3C%2FSPAN%3E%3CA%20href%3D%22https%3A%2F%2Fwww.youtube.com%2Fwatch%3Fv%3DGT4C84yrb68%22%20target%3D%22_blank%22%20rel%3D%22noopener%20nofollow%20noreferrer%22%3Ehttps%3A%2F%2Fwww.youtube.com%2Fwatch%3Fv%3DGT4C84yrb68%3C%2FA%3E%3C%2FLI%3E%0A%3CLI%3EScott%20Guthrie%20demoing%20Kusto%20in%20Techorama%3A%3CSPAN%3E%26nbsp%3B%3CBR%20%2F%3E%3C%2FSPAN%3E%3CA%20href%3D%22https%3A%2F%2Fwww.youtube.com%2Fwatch%3Fv%3DYTWewM_UMOk%26amp%3Bfeature%3Dyoutu.be%26amp%3Bt%3D3074%22%20target%3D%22_blank%22%20rel%3D%22noopener%20nofollow%20noreferrer%22%3Ehttps%3A%2F%2Fwww.youtube.com%2Fwatch%3Fv%3DYTWewM_UMOk%26amp%3Bfeature%3Dyoutu.be%26amp%3Bt%3D3074%3C%2FA%3E%3C%2FLI%3E%0A%3C%2FOL%3E%0A%3CP%3EKusto%20is%20used%20as%20the%20data%20platform%20for%20a%20number%20of%20Microsoft%20services%2C%20some%20of%20which%20expose%20its%20query%20language%20to%20users.%20Here%20are%20two%20videos%20showing%20its%20capabilities%20when%20used%20inside%20Application%20Insights%20%2F%20Azure%20Monitor%3A%3CBR%20%2F%3E%3CBR%20%2F%3E%3C%2FP%3E%0A%3COL%3E%0A%3CLI%3E%3CA%20href%3D%22https%3A%2F%2Fchannel9.msdn.com%2FEvents%2FBuild%2F2016%2FT666%3Focid%3Dplayer%22%20target%3D%22_blank%22%20rel%3D%22noopener%20noreferrer%22%3EInteractive%20Analytics%20with%20Application%20Insights%3C%2FA%3E%3C%2FLI%3E%0A%3CLI%3E%3CA%20href%3D%22https%3A%2F%2Fchannel9.msdn.com%2FEvents%2FBuild%2F2016%2FP591%3Focid%3Dplayer%22%20target%3D%22_blank%22%20rel%3D%22noopener%20noreferrer%22%3EAdvanced%20Analytics%20with%20Application%20Insights%3C%2FA%3E%3C%2FLI%3E%0A%3C%2FOL%3E%0A%3CP%3EProduct%20links%3A%3C%2FP%3E%0A%3CUL%3E%0A%3CLI%3EAzure%20Data%20Explorer%20%3CA%20href%3D%22https%3A%2F%2Faka.ms%2FAzureDataExplorer%22%20target%3D%22_blank%22%20rel%3D%22noopener%20noreferrer%22%3Ehttps%3A%2F%2Faka.ms%2FAzureDataExplorer%3C%2FA%3E%3C%2FLI%3E%0A%3CLI%3EAzure%20Data%20Explorer%20Docs%20%3CA%20href%3D%22https%3A%2F%2Faka.ms%2Fadx.docs%22%20target%3D%22_blank%22%20rel%3D%22noopener%20noreferrer%22%3Ehttps%3A%2F%2Faka.ms%2Fadx.docs%3C%2FA%3E%3C%2FLI%3E%0A%3CLI%3EMicrosoft%20Flow%20and%20Kusto%26nbsp%3B%3CA%20href%3D%22https%3A%2F%2Fflow.microsoft.com%2Fen-us%2Fconnectors%2Fshared_kusto%2Fazure-kusto%2F%22%20target%3D%22_blank%22%20rel%3D%22noopener%20noreferrer%22%3Ehttps%3A%2F%2Fflow.microsoft.com%2Fen-us%2Fconnectors%2Fshared_kusto%2Fazure-kusto%2F%3C%2FA%3E%3C%2FLI%3E%0A%3C%2FUL%3E%0A%3CP%3ESocial%3A%3C%2FP%3E%0A%3CUL%3E%0A%3CLI%3ETwitter%3A%3CSPAN%3E%26nbsp%3B%3C%2FSPAN%3E%3CA%20href%3D%22http%3A%2F%2Fwww.twitter.com%2FAzDataExplorer%22%20target%3D%22_self%22%20rel%3D%22nofollow%20noopener%20noreferrer%22%3E%3CSTRONG%3E%40AzDataExplorer%3C%2FSTRONG%3E%3C%2FA%3E%3C%2FLI%3E%0A%3CLI%3EStack%20Overflow%26nbsp%3B%3CA%20href%3D%22https%3A%2F%2Fstackoverflow.com%2Fsearch%3Fq%3DAzure%2BData%2BExplorer%22%20target%3D%22_blank%22%20rel%3D%22noopener%20nofollow%20noreferrer%22%3Ehttps%3A%2F%2Fstackoverflow.com%2Fsearch%3Fq%3DAzure%2BData%2BExplorer%3C%2FA%3E%3C%2FLI%3E%0A%3CLI%3ETechcommunity%20Blog%20for%20Kusto%26nbsp%3B%3CBR%20%2F%3E%3CA%20href%3D%22https%3A%2F%2Ftechcommunity.microsoft.com%2Ft5%2FAzure-Data-Explorer%2Fbd-p%2FKusto%22%20target%3D%22_blank%22%3Ehttps%3A%2F%2Ftechcommunity.microsoft.com%2Ft5%2FAzure-Data-Explorer%2Fbd-p%2FKusto%3C%2FA%3E%3C%2FLI%3E%0A%3C%2FUL%3E%3C%2FLINGO-BODY%3E%3CLINGO-TEASER%20id%3D%22lingo-teaser-739121%22%20slang%3D%22en-US%22%3E%3CP%3EAzure%20Data%20Explorer%20(Kusto)%20is%20a%20service%20for%20storing%20and%20running%20interactive%20analytics%20over%20Big%20Data.%3C%2FP%3E%0A%3CP%3EIt%20is%20based%20on%20relational%20database%20management%20systems%2C%20supporting%20entities%20such%20as%20databases%2C%20tables%2C%20and%20columns%2C%20as%20well%20as%20providing%20complex%20analytics%20query%20operators%20(such%20as%20calculated%20columns%2C%20searching%20and%20filtering%20or%20rows%2C%20group%20by-aggregates%2C%20joins).%3C%2FP%3E%0A%3CP%3E%26nbsp%3B%3C%2FP%3E%3C%2FLINGO-TEASER%3E
Version history
Last update:
‎Jul 05 2019 04:23 AM
Updated by: