Home
%3CLINGO-SUB%20id%3D%22lingo-sub-543690%22%20slang%3D%22en-US%22%3EWhat%E2%80%99s%20New%20with%20Azure%20HDInsight%20at%20%2F%2FBUILD%202019%3C%2FLINGO-SUB%3E%3CLINGO-BODY%20id%3D%22lingo-body-543690%22%20slang%3D%22en-US%22%3E%3CP%3E%26nbsp%3B%3C%2FP%3E%0A%3CP%3E%3CSTRONG%3E%3CSPAN%20class%3D%22lia-inline-image-display-wrapper%20lia-image-align-left%22%20style%3D%22width%3A%2096px%3B%22%3E%3CIMG%20src%3D%22https%3A%2F%2Fgxcuf89792.i.lithium.com%2Ft5%2Fimage%2Fserverpage%2Fimage-id%2F112501i78921AA06E1D6CF5%2Fimage-size%2Flarge%3Fv%3D1.0%26amp%3Bpx%3D999%22%20alt%3D%22HDI%20logo.png%22%20title%3D%22HDI%20logo.png%22%20%2F%3E%3C%2FSPAN%3E%3C%2FSTRONG%3E%3C%2FP%3E%0A%3CP%3E%26nbsp%3B%3C%2FP%3E%0A%3CP%3E%26nbsp%3B%3C%2FP%3E%0A%3CP%3E%26nbsp%3B%3C%2FP%3E%0A%3CP%3E%26nbsp%3B%3C%2FP%3E%0A%3CP%3E%3CSTRONG%3E%2F%2FBUILD%202019%2C%20SEATTLE%2C%20Washington%2C%20May%2008%2C%202019%3C%2FSTRONG%3E%20%E2%80%93%20Welcome%20to%20the%20%2F%2FBuild%20Conference%202019.%20Just%20a%20few%20short%20weeks%20ago%2C%20we%20announced%20the%20general%20availability%20of%20Apache%20Hadoop%203.0%20on%20Azure%20HDInsight%20(details%20%3CA%20href%3D%22https%3A%2F%2Fazure.microsoft.com%2Fen-us%2Fblog%2Fannouncing-general-availability-of-apache-hadoop-3-0-on-azure-hdinsight%2F%22%20target%3D%22_blank%22%20rel%3D%22noopener%20noopener%20noreferrer%20noopener%20noreferrer%22%3Ehere%3C%2FA%3E).%20Today%2C%20we%20are%20excited%20to%20bring%20several%20new%20features%20that%20will%20enable%20you%20to%20run%20your%20open-source%20analytics%20applications%20faster%2C%20cost%20lesser%20and%20easier%20to%20visualize%20and%20troubleshoot.%3C%2FP%3E%0A%3CP%3E%3CSTRONG%3E%3CU%3E%26nbsp%3B%3C%2FU%3E%3C%2FSTRONG%3E%3C%2FP%3E%0A%3CP%3E%3CSTRONG%3E%3CU%3ERun%20Faster%20with%20HBASE%20WRITE%20ACCELERATION%20(Preview)%3C%2FU%3E%3C%2FSTRONG%3E%3C%2FP%3E%0A%3CP%3EThe%20%3CA%20href%3D%22http%3A%2F%2Fhbase.apache.org%2Fbook.html%23purpose.wal%22%20target%3D%22_blank%22%20rel%3D%22noopener%20nofollow%20noopener%20noreferrer%20noopener%20noreferrer%22%3EWrite%20Ahead%20Log%3C%2FA%3E%20(WAL)%20in%20HBase%20plays%20a%20critical%20role%20in%20avoiding%20data%20consistency%20issues%20during%20Region%20Server%20crashes%20or%20unavailability.%20To%20ensure%20that%20there%20is%20no%20data%20loss%2C%20for%20every%20Put%2FDelete%20request%20in%20HBase%2C%20edits%20are%20written%20to%20WAL%20before%20the%20actual%20data%20is%20written%20to%20MemStore.%20In%20standard%20implementation%20of%20HDInsight%2C%20Write%20Ahead%20Logs%20(WALs)%20are%20stored%20as%20page%20blobs%20in%20Azure%20standard%20Storage.%20This%20works%20very%20well%20for%20most%20applications%2C%20but%20may%20not%20meet%20the%20needs%20of%20mission%20critical%20applications%20that%20require%20consistent%20low%20latency%20and%20high%20throughput%20I%2FO.%20Inconsistent%20latencies%20for%20WAL%20writes%20can%20be%20a%20bottleneck%20for%20overall%20write%20performance%20in%20HBase%20clusters.%20To%20improve%20the%20WAL%20write%20latencies%20in%20high%20performance%20clusters%2C%20we%20are%20introducing%20a%20new%20feature%20%E2%80%9CAccelerated%20Writes%E2%80%9D%20This%20feature%20attaches%20a%20Premium%20SSD%20Managed%20disk%20to%20every%20Region%20Server%20(worker%20node)%20%26amp%3B%20Write%20Ahead%20Log%20(WAL)%20is%20configured%20to%20be%20written%20to%20HDFS%20mounted%20on%20premium%20managed%20disks%20instead%20of%20standard%20Azure%20page%20blobs.%20Premium%20managed%20disks%20are%20SSD-based%20and%20offer%20excellent%20I%2FO%20performance%20with%20fault%20tolerance.%20As%20a%20result%2C%20customers%20will%20see%20low%20write%20latencies%20and%20better%20resiliency%20for%20HBase%20based%20workloads.%20To%20learn%20more%20please%20see%20%3CA%20href%3D%22https%3A%2F%2Fdocs.microsoft.com%2Fen-us%2Fazure%2Fhdinsight%2Fhbase%2Fapache-hbase-accelerated-writes%22%20target%3D%22_blank%22%20rel%3D%22noopener%20noopener%20noreferrer%20noopener%20noreferrer%22%3EHBase%20Accelerated%20Writes%3C%2FA%3E.%3C%2FP%3E%0A%3CP%3E%26nbsp%3B%3C%2FP%3E%0A%3CP%3E%3CSPAN%20class%3D%22lia-inline-image-display-wrapper%20lia-image-align-inline%22%20style%3D%22width%3A%20924px%3B%22%3E%3CIMG%20src%3D%22https%3A%2F%2Fgxcuf89792.i.lithium.com%2Ft5%2Fimage%2Fserverpage%2Fimage-id%2F112498i20E04CB1DEA240B3%2Fimage-size%2Flarge%3Fv%3D1.0%26amp%3Bpx%3D999%22%20alt%3D%22YCSB%20Perf%20Image.png%22%20title%3D%22YCSB%20Perf%20Image.png%22%20%2F%3E%3C%2FSPAN%3E%3C%2FP%3E%0A%3CP%3E%3CEM%3EYahoo%20Cloud%20Services%20Benchmark%3A%204%20D4V2%20region%20nodes%20with%20100-byte%20row%20size%20%3C%2FEM%3E%3C%2FP%3E%0A%3CP%3E%26nbsp%3B%3C%2FP%3E%0A%3CP%3E%3CSTRONG%3E%3CU%3EUse%20AutoScale%20to%20drive%20higher%20utilization%20at%20a%20lower%20cost%20(Preview)%3C%2FU%3E%3C%2FSTRONG%3E%3C%2FP%3E%0A%3CP%3EHDInsight%20Autoscale%20enables%20enterprises%20to%20become%20more%20productive%20and%20cost-efficient%20by%20automatically%20scaling%20clusters%20up%20or%20down%20based%20on%20the%20load%20or%20a%20customized%20schedule.%26nbsp%3BHDInsight%20Autoscale%20allows%20you%20to%20customize%20the%20scaling%20strategy%20based%20on%20your%20own%20scenario.%20You%20can%20either%20define%20the%20maximum%20and%20minimum%20based%20on%20your%20cost%20requirements%20or%20define%20a%20schedule%20for%20each%20weekday%20to%20meet%20your%20own%20business%20objectives.%20After%20that%2C%20you%20can%20rest%20assured%20right%20scaling%20decision%20will%20be%20made%20and%20you%20will%20pay%20only%20what%20you%20need.%20To%20learn%20more%2C%20check%20out%20the%20documentation%20%3CA%20href%3D%22https%3A%2F%2Fdocs.microsoft.com%2Fen-us%2Fazure%2Fhdinsight%2Fhdinsight-autoscale-clusters%22%20target%3D%22_blank%22%20rel%3D%22noopener%20noopener%20noreferrer%20noopener%20noreferrer%22%3Ehere%3C%2FA%3E.%3C%2FP%3E%0A%3CP%3E%26nbsp%3B%3C%2FP%3E%0A%3CP%3E%3CSTRONG%3E%3CU%3EVisualize%20Spark%20jobs%20and%20data%20in%20Jupyter%20%3C%2FU%3E%3C%2FSTRONG%3E%3C%2FP%3E%0A%3CP%3EData%20scientists%20use%20Jupyter%20Notebooks%20on%20HDInsight%20Spark%20to%20quickly%20explore%20data%20sets%2C%20perform%20trend%20analysis%2C%20or%20try%20different%20machine%20learning%20models.%20Without%20the%20ability%20to%20track%20the%20status%20of%20Spark%20jobs%20and%20intermediate%20data%2C%20data%20scientists%20find%20it%20difficult%20to%20monitor%20and%20optimize%20their%20queries.%3C%2FP%3E%0A%3CP%3EJupyter%20Notebooks%20on%20Azure%20HDInsight%20now%20enables%20cutting-edge%20job%20execution%20and%20visualization%20experiences%20including%20Spark%20job%20progress%20and%20cell%20execution%20status%20indicators%20and%20native%20matplotlib%20support%20for%20PySpark%20DataFrame.%20To%20learn%20more%2C%20please%20see%20our%20blog%20%3CA%20href%3D%22https%3A%2F%2Fazure.microsoft.com%2Fen-us%2Fblog%2Funderstanding-hdinsight-spark-jobs-and-data-through-visualizations-in-the-jupyter-notebook%2F%22%20target%3D%22_blank%22%20rel%3D%22noopener%20noopener%20noreferrer%20noopener%20noreferrer%22%3Ehere%3C%2FA%3E.%3C%2FP%3E%0A%3CP%3E%26nbsp%3B%3C%2FP%3E%0A%3CP%3E%3CSTRONG%3E%3CU%3EHDInsight%20at%20%2F%2FBuild%202019%3C%2FU%3E%3C%2FSTRONG%3E%3C%2FP%3E%0A%3CP%3EThe%20HDInsight%20team%20will%20be%20available%20at%20%2F%2FBUILD%202019%20during%20all%20three%20days%20of%20the%20conference.%20Please%20take%20the%20time%20to%20come%20by%20and%20meet%20the%20team%20at%20the%20HDInsight%20booth%20on%20the%20Expo%20floor.%20We%20would%20be%20happy%20to%20show%20you%20more%20of%20what%20we%20have%20to%20offer%20and%20answer%20any%20questions%20you%20might%20have.%20And%2C%20while%20you%20are%20at%20%2F%2FBUILD%2C%20do%20not%20miss%20the%20following%20sessions%20and%20demos%3A%3C%2FP%3E%0A%3CUL%3E%0A%3CLI%3EData%20Architect%E2%80%99s%20guide%20for%20successful%20Open%20Source%20patterns%20in%20Azure%20(%3CA%20href%3D%22https%3A%2F%2Fmybuild.techcommunity.microsoft.com%2Fsessions%2F77001%3Fsource%3Dsessions%23top-anchor%22%20target%3D%22_blank%22%20rel%3D%22noopener%20noopener%20noreferrer%20noopener%20noreferrer%22%3Elink%3C%2FA%3E)%20on%20May%208%3CSUP%3Eth%3C%2FSUP%3E%2C%202019%20%40%205pm%20PT.%3C%2FLI%3E%0A%3CLI%3EDemo%3A%20Monitoring%20hundreds%20of%20HDInsight%20clusters%20with%20Azure%20Log%20Analytics%3C%2FLI%3E%0A%3CLI%3EDemo%3A%20Data%20Accelerator%20on%20Azure%20HDInsight%3C%2FLI%3E%0A%3C%2FUL%3E%0A%3CP%3E%26nbsp%3B%3C%2FP%3E%0A%3CP%3E%3CSTRONG%3E%3CU%3EGet%20Started%20Now%3C%2FU%3E%3C%2FSTRONG%3E%3C%2FP%3E%0A%3CP%3EWe%20are%20excited%20to%20see%20what%20you%20will%20build%20next%20with%20Azure%20HDInsight.%20%3CA%20href%3D%22https%3A%2F%2Fazure.microsoft.com%2Fen-us%2Fblog%2Fget-up-to-speed-with-azure-hdinsight-the-comprehensive-guide%2F%22%20target%3D%22_blank%22%20rel%3D%22noopener%20noopener%20noreferrer%20noopener%20noreferrer%22%3ERead%20this%20developer%20guide%3C%2FA%3E%20and%20follow%20the%20%3CA%20href%3D%22https%3A%2F%2Fdocs.microsoft.com%2Fen-us%2Fazure%2Fhdinsight%2Fhadoop%2Fapache-hadoop-linux-tutorial-get-started%22%20target%3D%22_blank%22%20rel%3D%22noopener%20noopener%20noreferrer%20noopener%20noreferrer%22%3Equick%20start%20guide%3C%2FA%3E%20to%20learn%20more%20about%20implementing%20open%20source%20analytics%20pipelines%20on%20Azure%20HDInsight.%20Stay%20up-to-date%20on%20the%20latest%20Azure%20HDInsight%20news%20and%20features%20coming%20up%20in%20the%20near%20future%20by%20following%20us%20on%20Twitter%20%3CA%20href%3D%22https%3A%2F%2Ftwitter.com%2Fsearch%3Fq%3D%2523HDInsight%26amp%3Bsrc%3Dtypd%22%20target%3D%22_blank%22%20rel%3D%22noopener%20nofollow%20noopener%20noreferrer%20noopener%20noreferrer%22%3E%23HDInsight%3C%2FA%3E%20and%20%3CA%20href%3D%22https%3A%2F%2Ftwitter.com%2FAzureHDInsight%22%20target%3D%22_blank%22%20rel%3D%22noopener%20nofollow%20noopener%20noreferrer%20noopener%20noreferrer%22%3E%40AzureHDInsight%3C%2FA%3E.%20For%20questions%20and%20feedback%2C%20please%20reach%20out%20to%20AskHDInsight%40microsoft.com.%3C%2FP%3E%0A%3CP%3E%26nbsp%3B%3C%2FP%3E%0A%3CP%3E%3CSTRONG%3E%3CU%3EAbout%20Azure%20HDInsight%3C%2FU%3E%3C%2FSTRONG%3E%3C%2FP%3E%0A%3CP%3EAzure%20HDInsight%20is%20an%20enterprise-ready%20platform%20for%20open%20source%20analytics%20that%20enables%20customers%20to%20easily%20run%20popular%20Apache%20open%20source%20frameworks%20including%20Apache%20Hadoop%2C%20Spark%2C%20Kafka%2C%20and%20others.%20Brought%20to%20you%20in%20close%20partnership%20with%20Cloudera%2C%20the%20service%20is%20available%20in%2030%20public%20regions%20and%20Azure%20Government%20Clouds%20in%20the%20US%20and%20Germany.%20Azure%20HDInsight%20powers%20mission%20critical%20applications%20for%20a%20wide%20range%20of%20sectors%20and%20use%20cases%20including%20ETL%2C%20streaming%2C%20and%20interactive%20querying.%3C%2FP%3E%3C%2FLINGO-BODY%3E%3CLINGO-TEASER%20id%3D%22lingo-teaser-543690%22%20slang%3D%22en-US%22%3E%3CP%3EJust%20a%20few%20short%20weeks%20ago%2C%20we%20announced%20the%20general%20availability%20of%20Apache%20Hadoop%203.0%20on%20Azure%20HDInsight%20(details%20here).%20We%20are%20now%20excited%20to%20bring%20several%20new%20features%20that%20will%20enable%20you%20to%20run%20your%20open-source%20analytics%20applications%20faster%2C%20cost%20lesser%20and%20easier%20to%20visualize%20and%20troubleshoot.%3C%2FP%3E%3C%2FLINGO-TEASER%3E

 

HDI logo.png

 

 

 

 

//BUILD 2019, SEATTLE, Washington, May 08, 2019 – Welcome to the //Build Conference 2019. Just a few short weeks ago, we announced the general availability of Apache Hadoop 3.0 on Azure HDInsight (details here). Today, we are excited to bring several new features that will enable you to run your open-source analytics applications faster, cost lesser and easier to visualize and troubleshoot.

 

Run Faster with HBASE WRITE ACCELERATION (Preview)

The Write Ahead Log (WAL) in HBase plays a critical role in avoiding data consistency issues during Region Server crashes or unavailability. To ensure that there is no data loss, for every Put/Delete request in HBase, edits are written to WAL before the actual data is written to MemStore. In standard implementation of HDInsight, Write Ahead Logs (WALs) are stored as page blobs in Azure standard Storage. This works very well for most applications, but may not meet the needs of mission critical applications that require consistent low latency and high throughput I/O. Inconsistent latencies for WAL writes can be a bottleneck for overall write performance in HBase clusters. To improve the WAL write latencies in high performance clusters, we are introducing a new feature “Accelerated Writes” This feature attaches a Premium SSD Managed disk to every Region Server (worker node) & Write Ahead Log (WAL) is configured to be written to HDFS mounted on premium managed disks instead of standard Azure page blobs. Premium managed disks are SSD-based and offer excellent I/O performance with fault tolerance. As a result, customers will see low write latencies and better resiliency for HBase based workloads. To learn more please see HBase Accelerated Writes.

 

YCSB Perf Image.png

Yahoo Cloud Services Benchmark: 4 D4V2 region nodes with 100-byte row size

 

Use AutoScale to drive higher utilization at a lower cost (Preview)

HDInsight Autoscale enables enterprises to become more productive and cost-efficient by automatically scaling clusters up or down based on the load or a customized schedule. HDInsight Autoscale allows you to customize the scaling strategy based on your own scenario. You can either define the maximum and minimum based on your cost requirements or define a schedule for each weekday to meet your own business objectives. After that, you can rest assured right scaling decision will be made and you will pay only what you need. To learn more, check out the documentation here.

 

Visualize Spark jobs and data in Jupyter

Data scientists use Jupyter Notebooks on HDInsight Spark to quickly explore data sets, perform trend analysis, or try different machine learning models. Without the ability to track the status of Spark jobs and intermediate data, data scientists find it difficult to monitor and optimize their queries.

Jupyter Notebooks on Azure HDInsight now enables cutting-edge job execution and visualization experiences including Spark job progress and cell execution status indicators and native matplotlib support for PySpark DataFrame. To learn more, please see our blog here.

 

HDInsight at //Build 2019

The HDInsight team will be available at //BUILD 2019 during all three days of the conference. Please take the time to come by and meet the team at the HDInsight booth on the Expo floor. We would be happy to show you more of what we have to offer and answer any questions you might have. And, while you are at //BUILD, do not miss the following sessions and demos:

  • Data Architect’s guide for successful Open Source patterns in Azure (link) on May 8th, 2019 @ 5pm PT.
  • Demo: Monitoring hundreds of HDInsight clusters with Azure Log Analytics
  • Demo: Data Accelerator on Azure HDInsight

 

Get Started Now

We are excited to see what you will build next with Azure HDInsight. Read this developer guide and follow the quick start guide to learn more about implementing open source analytics pipelines on Azure HDInsight. Stay up-to-date on the latest Azure HDInsight news and features coming up in the near future by following us on Twitter #HDInsight and @AzureHDInsight. For questions and feedback, please reach out to AskHDInsight@microsoft.com.

 

About Azure HDInsight

Azure HDInsight is an enterprise-ready platform for open source analytics that enables customers to easily run popular Apache open source frameworks including Apache Hadoop, Spark, Kafka, and others. Brought to you in close partnership with Cloudera, the service is available in 30 public regions and Azure Government Clouds in the US and Germany. Azure HDInsight powers mission critical applications for a wide range of sectors and use cases including ETL, streaming, and interactive querying.