Speed up your data workloads with performance updates to Apache Spark 3.1.2 in Azure Synapse

Published Sep 21 2021 12:42 PM 3,398 Views
Microsoft

The Azure Synapse Analytics team is continually focused on delivering a highly performant and scalable platform for Apache Spark workloads, in particular in support of our customers' most common workload patterns. By combining the latest open-source updates in Apache Spark with a strong focus on optimizing performance, customers can now take advantage of significant performance gains reflected in the below benchmarking tests1

  Aria_Jelinek_0-1632499257797.png

 

We have recently announced General Availability of Apache Spark 3.1.2 as a part of Azure Synapse Analytics. This delivers significant performance improvements over Apache Spark 2.4. In the new release of Spark on Azure Synapse Analytics, our benchmark performance tests indicate that we have also been able to achieve a 13% improvement in performance from the previous release and run 202% faster than Apache Spark 3.1.2. This means you can do more with your data, faster and at a lower cost.

 

Aria_Jelinek_1-1632499489521.png

 

Previously, we improved Apache Spark performance through query optimization, autoscaling, cluster optimizations, intelligent caching, and indexing. In the latest release, we have further improved Apache Spark 3.1.2 in Azure Synapse Analytics by using the following three optimizations:

 

  • Limit pushdown
  • Optimized sorts
  • Bloom filter enhancements

Limit Pushdown

This optimization applies while performing top-k queries by eliminating compute cycles involved in processing rows which are not part of the top-k within the partition.

 

For example, when identifying the top-selling products across categories, where data is partitioned by categories, identifying the top-k rows within a shuffle, and comparing just those that fall within the top-k across partitions will eliminate the need for processing other rows.

 

Statistics must be enabled to trigger this optimization.

 

Optimized Sort

Sorting is one of the most used and computationally expensive operations along with aggregations. In Synapse Spark, we have written an optimized an implementation of sorting which benefits from prior partitioning of data.

 

This new algorithm can leverage cardinality information to create multiple sorters and efficiently use prefix comparison. Prefix comparisons are way faster than record comparisons. For sorting on multiple columns, we reorder sorting columns to reduce the number of record comparisons required.

 

This is very useful for queries requiring window operation like getting top 100 highly paid employees in each department or getting 100 most selling products in different categories.

 

Bloom filter enhancements

In this release, we have extended support of Bloom filters to sort merge joins in addition to broadcast hash joins which we talked about previously.

 

Shuffling is a bottleneck in query execution as it requires data to be written on the disk. We have further enhanced Bloom filter implementation in Synapse Spark to operate on sort merge joins. The idea is to create Bloom filters from the smaller tables and leverage them to prune large tables. This will help in reducing shuffle data and thus improving query performance. With this extension we were able to reduce shuffle sizes by 50%.

 

For example, given a fact table ‘Sales’ and a dimension table ‘Items’, application of a Bloom filter will drastically improve performance when we want to get total sales for selected items.

 

Limit push-down and optimized sort are effective when stats are enabled & the Analyze Table command is run.

 

Summary

Continuous improvements to performance tuning and optimizations in Azure Synapse Analytics enable you to run your Apache Spark workloads in a cost-effective manner with reduced processing times.

The workload used was derived from the TPC-DS benchmark but did not follow the formal benchmarking process. As such, this workload's results are not comparable to published TPC-DS results.

5 Comments
Senior Member

Hi @balajisankaran ,

You wrote 

We have recently announced General Availability of Apache Spark 3.1.2

Can you please refer to the announcement / more details?

I thought that Spark 2.4 is GA and Spark 3.1 is in Public Preview.

Microsoft
Senior Member

Thanks @balajisankaran , so the documentation is not up to date?

 

One more question: do Data Flows (as activity within Synapse Pipelines) also benefit from the improvements and will run on a Spark 3.1 cluster instead of 2.4?

Regular Visitor

It does not support querying s

Delta table by version, we can't create delta as  external table on dedicated  pooletc. Long way to compete with databricks, no optimize or zorder type functionality  as well.

Microsoft

@_MartinB  Data Flows currently only runs on Spark 2.4

%3CLINGO-SUB%20id%3D%22lingo-sub-2773336%22%20slang%3D%22en-US%22%3ERe%3A%20Performance%20Improvements%20to%20Apache%20Spark%203.1.2%20in%20Azure%20Synapse%3C%2FLINGO-SUB%3E%3CLINGO-BODY%20id%3D%22lingo-body-2773336%22%20slang%3D%22en-US%22%3E%3CP%3EHi%20%3CA%20href%3D%22https%3A%2F%2Ftechcommunity.microsoft.com%2Ft5%2Fuser%2Fviewprofilepage%2Fuser-id%2F1148462%22%20target%3D%22_blank%22%3E%40balajisankaran%3C%2FA%3E%26nbsp%3B%2C%3C%2FP%3E%3CP%3EYou%20wrote%26nbsp%3B%3C%2FP%3E%3CBLOCKQUOTE%3E%3CP%3E%3CSPAN%3EWe%20have%20recently%20announced%20General%20Availability%20of%20Apache%20Spark%203.1.2%3C%2FSPAN%3E%3C%2FP%3E%3C%2FBLOCKQUOTE%3E%3CP%3ECan%20you%20please%20refer%20to%20the%20announcement%20%2F%20more%20details%3F%3C%2FP%3E%3CP%3EI%20thought%20that%20Spark%202.4%20is%20GA%20and%20%3CA%20href%3D%22https%3A%2F%2Fwww.google.com%2Furl%3Fsa%3Dt%26amp%3Bsource%3Dweb%26amp%3Brct%3Dj%26amp%3Burl%3Dhttps%3A%2F%2Fdocs.microsoft.com%2Fen-us%2Fazure%2Fsynapse-analytics%2Fspark%2Fapache-spark-3-runtime%26amp%3Bved%3D2ahUKEwi_l9OWp5LzAhUKyIUKHZ0yCW4QFnoECA0QAQ%26amp%3Busg%3DAOvVaw2urrKWqR3YJX4C1BgJkMPc%22%20target%3D%22_self%22%20rel%3D%22nofollow%20noopener%20noreferrer%22%3ESpark%203.1%20is%20in%20Public%20Preview%3C%2FA%3E.%3C%2FP%3E%3C%2FLINGO-BODY%3E%3CLINGO-SUB%20id%3D%22lingo-sub-2773340%22%20slang%3D%22en-US%22%3ERe%3A%20Performance%20Improvements%20to%20Apache%20Spark%203.1.2%20in%20Azure%20Synapse%3C%2FLINGO-SUB%3E%3CLINGO-BODY%20id%3D%22lingo-body-2773340%22%20slang%3D%22en-US%22%3E%3CP%3E%3CA%20href%3D%22https%3A%2F%2Ftechcommunity.microsoft.com%2Ft5%2Fuser%2Fviewprofilepage%2Fuser-id%2F939272%22%20target%3D%22_blank%22%3E%40_MartinB%3C%2FA%3E%26nbsp%3B-%3C%2FP%3E%0A%3CP%3E%26nbsp%3B%3C%2FP%3E%0A%3CP%3EYou%20can%20find%20the%20release%20details%20here%20-%26nbsp%3B%3CA%20href%3D%22https%3A%2F%2Fazure.microsoft.com%2Fen-us%2Fupdates%2Fapache-spark-31-for-azure-synapse-analytics-now-generally-available%2F%22%20target%3D%22_blank%22%20rel%3D%22noopener%20noreferrer%22%3EApache%20Spark%203.1%20for%20Azure%20Synapse%20Analytics%20now%20generally%20available%20%7C%20Azure%20updates%20%7C%20Microsoft%20Azure%3C%2FA%3E%3C%2FP%3E%3C%2FLINGO-BODY%3E%3CLINGO-SUB%20id%3D%22lingo-sub-2775517%22%20slang%3D%22en-US%22%3ERe%3A%20Performance%20Improvements%20to%20Apache%20Spark%203.1.2%20in%20Azure%20Synapse%3C%2FLINGO-SUB%3E%3CLINGO-BODY%20id%3D%22lingo-body-2775517%22%20slang%3D%22en-US%22%3E%3CP%3EIt%20does%20not%20support%20querying%20s%3C%2FP%3E%3CP%3EDelta%20table%20by%20version%2C%20we%20can't%20create%20delta%20as%26nbsp%3B%20external%20table%20on%20dedicated%26nbsp%3B%20pooletc.%20Long%20way%20to%20compete%20with%20databricks%2C%20no%20optimize%20or%20zorder%20type%20functionality%26nbsp%3B%20as%20well.%3C%2FP%3E%3C%2FLINGO-BODY%3E%3CLINGO-SUB%20id%3D%22lingo-sub-2777011%22%20slang%3D%22en-US%22%3ERe%3A%20Performance%20Improvements%20to%20Apache%20Spark%203.1.2%20in%20Azure%20Synapse%3C%2FLINGO-SUB%3E%3CLINGO-BODY%20id%3D%22lingo-body-2777011%22%20slang%3D%22en-US%22%3E%3CP%3E%3CA%20href%3D%22https%3A%2F%2Ftechcommunity.microsoft.com%2Ft5%2Fuser%2Fviewprofilepage%2Fuser-id%2F939272%22%20target%3D%22_blank%22%3E%40_MartinB%3C%2FA%3E%26nbsp%3B%3CSPAN%3E%26nbsp%3BData%20Flows%20currently%20only%20runs%20on%20Spark%202.4%3C%2FSPAN%3E%3C%2FP%3E%3C%2FLINGO-BODY%3E%3CLINGO-SUB%20id%3D%22lingo-sub-2769467%22%20slang%3D%22en-US%22%3ESpeed%20up%20your%20data%20workloads%20with%20performance%20updates%20to%20Apache%20Spark%203.1.2%20in%20Azure%20Synapse%3C%2FLINGO-SUB%3E%3CLINGO-BODY%20id%3D%22lingo-body-2769467%22%20slang%3D%22en-US%22%3E%3CP%3EThe%20Azure%20Synapse%20Analytics%20team%20is%20continually%20focused%20on%20delivering%20a%20highly%20performant%20and%20scalable%20platform%20for%20Apache%20Spark%20workloads%2C%20in%20particular%20in%20support%20of%20our%20customers'%20most%20common%20workload%20patterns.%20By%20combining%20the%20latest%20open-source%20updates%20in%20Apache%20Spark%20with%20a%20strong%20focus%20on%20optimizing%20performance%2C%20customers%20can%20now%20take%20advantage%20of%20significant%20performance%20gains%20reflected%20in%20the%20below%20benchmarking%20tests%3CSUP%3E1%3C%2FSUP%3E%3C%2FP%3E%0A%3CP%3E%26nbsp%3B%26nbsp%3B%3CSPAN%20class%3D%22lia-inline-image-display-wrapper%20lia-image-align-center%22%20image-alt%3D%22Aria_Jelinek_0-1632499257797.png%22%20style%3D%22width%3A%20522px%3B%22%3E%3CIMG%20src%3D%22https%3A%2F%2Ftechcommunity.microsoft.com%2Ft5%2Fimage%2Fserverpage%2Fimage-id%2F312723i0D1C6900D239D0C0%2Fimage-dimensions%2F522x318%3Fv%3Dv2%22%20width%3D%22522%22%20height%3D%22318%22%20role%3D%22button%22%20title%3D%22Aria_Jelinek_0-1632499257797.png%22%20alt%3D%22Aria_Jelinek_0-1632499257797.png%22%20%2F%3E%3C%2FSPAN%3E%3C%2FP%3E%0A%3CP%3E%26nbsp%3B%3C%2FP%3E%0A%3CP%3EWe%20have%20recently%20announced%20%3CA%20href%3D%22https%3A%2F%2Fazure.microsoft.com%2Fen-in%2Fupdates%2Fapache-spark-31-for-azure-synapse-analytics-now-generally-available%2F%22%20target%3D%22_blank%22%20rel%3D%22noopener%20noreferrer%22%3EGeneral%20Availability%20of%20Apache%20Spark%203.1.2%3C%2FA%3E%20as%20a%20part%20of%20Azure%20Synapse%20Analytics.%20This%20delivers%20significant%20performance%20improvements%20over%26nbsp%3BApache%20Spark%202.4.%20In%20the%20new%20release%20of%20Spark%20on%20Azure%20Synapse%20Analytics%2C%20our%20benchmark%20performance%20tests%20indicate%20that%20we%20have%20also%20been%20able%20to%20achieve%20a%2013%25%20improvement%20in%20performance%20from%20the%20previous%20release%20and%20run%20202%25%20faster%20than%20Apache%20Spark%203.1.2.%20%3CSTRONG%3EThis%20means%20you%20can%20do%20more%20with%20your%20data%2C%20faster%20and%20at%20a%20lower%20cost%3C%2FSTRONG%3E.%3C%2FP%3E%0A%3CP%3E%26nbsp%3B%3C%2FP%3E%0A%3CP%3E%3CSPAN%20class%3D%22lia-inline-image-display-wrapper%20lia-image-align-center%22%20image-alt%3D%22Aria_Jelinek_1-1632499489521.png%22%20style%3D%22width%3A%20762px%3B%22%3E%3CIMG%20src%3D%22https%3A%2F%2Ftechcommunity.microsoft.com%2Ft5%2Fimage%2Fserverpage%2Fimage-id%2F312725i2FAE9CC07AD3C381%2Fimage-dimensions%2F762x240%3Fv%3Dv2%22%20width%3D%22762%22%20height%3D%22240%22%20role%3D%22button%22%20title%3D%22Aria_Jelinek_1-1632499489521.png%22%20alt%3D%22Aria_Jelinek_1-1632499489521.png%22%20%2F%3E%3C%2FSPAN%3E%3C%2FP%3E%0A%3CP%3E%26nbsp%3B%3C%2FP%3E%0A%3CP%3E%3CA%20href%3D%22https%3A%2F%2Ftechcommunity.microsoft.com%2Ft5%2Fazure-synapse-analytics%2Fapache-spark-in-azure-synapse-performance-update%2Fba-p%2F2243534%22%20target%3D%22_blank%22%3EPreviously%3C%2FA%3E%2C%20we%20improved%20Apache%20Spark%20performance%20through%20query%20optimization%2C%20autoscaling%2C%20cluster%20optimizations%2C%20intelligent%20caching%2C%20and%20indexing.%20In%20the%20latest%20release%2C%20we%20have%20further%20improved%20Apache%20Spark%203.1.2%26nbsp%3Bin%20Azure%20Synapse%20Analytics%20by%20using%20the%20following%20three%20optimizations%3A%3C%2FP%3E%0A%3CP%3E%26nbsp%3B%3C%2FP%3E%0A%3CUL%3E%0A%3CLI%3ELimit%20pushdown%3C%2FLI%3E%0A%3CLI%3EOptimized%20sorts%3C%2FLI%3E%0A%3CLI%3EBloom%20filter%20enhancements%3CBR%20%2F%3E%3CBR%20%2F%3E%3C%2FLI%3E%0A%3C%2FUL%3E%0A%3CP%3E%3CSTRONG%3ELimit%20Pushdown%3C%2FSTRONG%3E%3C%2FP%3E%0A%3CP%3EThis%20optimization%20applies%20while%20performing%20top-k%20queries%20by%20eliminating%20compute%20cycles%20involved%20in%20processing%20rows%20which%20are%20not%20part%20of%20the%20top-k%20within%20the%20partition.%3C%2FP%3E%0A%3CP%3E%26nbsp%3B%3C%2FP%3E%0A%3CP%3EFor%20example%2C%20when%20identifying%20the%20top-selling%20products%20across%20categories%2C%20where%20data%20is%20partitioned%20by%20categories%2C%20identifying%20the%20top-k%20rows%20within%20a%20shuffle%2C%20and%20comparing%20just%20those%20that%20fall%20within%20the%20top-k%20across%20partitions%20will%20eliminate%20the%20need%20for%20processing%20other%20rows.%3C%2FP%3E%0A%3CP%3E%26nbsp%3B%3C%2FP%3E%0A%3CP%3EStatistics%20must%20be%20enabled%20to%20trigger%20this%20optimization.%3C%2FP%3E%0A%3CP%3E%26nbsp%3B%3C%2FP%3E%0A%3CP%3E%3CSTRONG%3EOptimized%20Sort%3C%2FSTRONG%3E%3C%2FP%3E%0A%3CP%3ESorting%20is%20one%20of%20the%20most%20used%20and%20computationally%20expensive%20operations%20along%20with%20aggregations.%20In%20Synapse%20Spark%2C%20we%20have%20written%20an%20optimized%20an%20implementation%20of%20sorting%20which%20benefits%20from%20prior%20partitioning%20of%20data.%3C%2FP%3E%0A%3CP%3E%26nbsp%3B%3C%2FP%3E%0A%3CP%3EThis%20new%20algorithm%20can%20leverage%20cardinality%20information%20to%20create%20multiple%20sorters%20and%20efficiently%20use%20prefix%20comparison.%20Prefix%20comparisons%20are%20way%20faster%20than%20record%20comparisons.%20For%20sorting%20on%20multiple%20columns%2C%20we%20reorder%20sorting%20columns%20to%20reduce%20the%20number%20of%20record%20comparisons%20required.%3C%2FP%3E%0A%3CP%3E%26nbsp%3B%3C%2FP%3E%0A%3CP%3EThis%20is%20very%20useful%20for%20queries%20requiring%20window%20operation%20like%20getting%20top%20100%20highly%20paid%20employees%20in%20each%20department%20or%20getting%20100%20most%20selling%20products%20in%20different%20categories.%3C%2FP%3E%0A%3CP%3E%26nbsp%3B%3C%2FP%3E%0A%3CP%3E%3CSTRONG%3EBloom%20filter%20enhancements%3C%2FSTRONG%3E%3C%2FP%3E%0A%3CP%3EIn%20this%20release%2C%20we%20have%20extended%20support%20of%20Bloom%20filters%20to%20sort%20merge%20joins%20in%20addition%20to%20broadcast%20hash%20joins%20which%20we%20talked%20about%20%3CA%20href%3D%22https%3A%2F%2Ftechcommunity.microsoft.com%2Ft5%2Fazure-synapse-analytics%2Fapache-spark-in-azure-synapse-performance-update%2Fba-p%2F2243534%22%20target%3D%22_blank%22%3Ep%3C%2FA%3E%3CA%20href%3D%22https%3A%2F%2Ftechcommunity.microsoft.com%2Ft5%2Fazure-synapse-analytics%2Fapache-spark-in-azure-synapse-performance-update%2Fba-p%2F2243534%22%20target%3D%22_blank%22%3Ereviously%3C%2FA%3E.%3C%2FP%3E%0A%3CP%3E%26nbsp%3B%3C%2FP%3E%0A%3CP%3EShuffling%20is%20a%20bottleneck%20in%20query%20execution%20as%20it%20requires%20data%20to%20be%20written%20on%20the%20disk.%20We%20have%20further%20enhanced%20Bloom%20filter%20implementation%20in%20Synapse%20Spark%20to%20operate%20on%20sort%20merge%20joins.%20The%20idea%20is%20to%20create%20Bloom%20filters%20from%20the%20smaller%20tables%20and%20leverage%20them%20to%20prune%20large%20tables.%20This%20will%20help%20in%20reducing%26nbsp%3Bshuffle%20data%26nbsp%3Band%20thus%20improving%20query%20performance.%20With%20this%20extension%20we%20were%20able%20to%20reduce%20shuffle%20sizes%20by%2050%25.%3C%2FP%3E%0A%3CP%3E%26nbsp%3B%3C%2FP%3E%0A%3CP%3EFor%20example%2C%20given%20a%20fact%20table%20%E2%80%98Sales%E2%80%99%20and%20a%20dimension%20table%20%E2%80%98Items%E2%80%99%2C%20application%20of%20a%20Bloom%20filter%20will%20drastically%20improve%20performance%20when%20we%20want%20to%20get%20total%20sales%20for%20selected%20items.%3C%2FP%3E%0A%3CP%3E%26nbsp%3B%3C%2FP%3E%0A%3CP%3ELimit%20push-down%20and%20optimized%20sort%20are%20effective%20when%20stats%20are%20enabled%20%26amp%3B%20the%20%3CA%20href%3D%22https%3A%2F%2Fspark.apache.org%2Fdocs%2F3.0.0-preview%2Fsql-ref-syntax-aux-analyze-table.html%22%20target%3D%22_blank%22%20rel%3D%22noopener%20nofollow%20noreferrer%22%3EAnalyze%20Table%3C%2FA%3E%20command%20is%20run.%3C%2FP%3E%0A%3CP%3E%26nbsp%3B%3C%2FP%3E%0A%3CP%3E%3CSTRONG%3ESummary%3C%2FSTRONG%3E%3C%2FP%3E%0A%3CP%3EContinuous%20improvements%20to%20performance%20tuning%20and%20optimizations%20in%20Azure%20Synapse%20Analytics%20enable%20you%20to%20run%20your%20Apache%20Spark%20workloads%20in%20a%20cost-effective%20manner%20with%20reduced%20processing%20times.%3CBR%20%2F%3E%3CBR%20%2F%3E%3C%2FP%3E%0A%3CP%3E1%26nbsp%3B%3CSPAN%3EThe%20workload%20used%20was%20derived%20from%20the%20TPC-DS%20benchmark%20but%20did%20not%20follow%20the%20formal%20benchmarking%20process.%20As%20such%2C%20this%20workload's%20results%20are%20not%20comparable%20to%20published%20TPC-DS%20results.%3C%2FSPAN%3E%3C%2FP%3E%3C%2FLINGO-BODY%3E%3CLINGO-TEASER%20id%3D%22lingo-teaser-2769467%22%20slang%3D%22en-US%22%3E%3CP%3E%3CSPAN%20class%3D%22lia-inline-image-display-wrapper%20lia-image-align-inline%22%20image-alt%3D%22balajisankaran_0-1632203040804.png%22%20style%3D%22width%3A%20400px%3B%22%3E%3CIMG%20src%3D%22https%3A%2F%2Ftechcommunity.microsoft.com%2Ft5%2Fimage%2Fserverpage%2Fimage-id%2F311742i9DBCE2AFC0F8C3C0%2Fimage-size%2Fmedium%3Fv%3Dv2%26amp%3Bpx%3D400%22%20role%3D%22button%22%20title%3D%22balajisankaran_0-1632203040804.png%22%20alt%3D%22balajisankaran_0-1632203040804.png%22%20%2F%3E%3C%2FSPAN%3E%3C%2FP%3E%0A%3CP%3EApache%20Spark%203.1.2%20performance%20in%20Azure%20Synapse%20gets%20even%20faster!%3C%2FP%3E%3C%2FLINGO-TEASER%3E%3CLINGO-LABS%20id%3D%22lingo-labs-2769467%22%20slang%3D%22en-US%22%3E%3CLINGO-LABEL%3ESynapse%20Spark%3C%2FLINGO-LABEL%3E%3CLINGO-LABEL%3EUpdates%3C%2FLINGO-LABEL%3E%3C%2FLINGO-LABS%3E%3CLINGO-SUB%20id%3D%22lingo-sub-2773373%22%20slang%3D%22en-US%22%3ERe%3A%20Performance%20Improvements%20to%20Apache%20Spark%203.1.2%20in%20Azure%20Synapse%3C%2FLINGO-SUB%3E%3CLINGO-BODY%20id%3D%22lingo-body-2773373%22%20slang%3D%22en-US%22%3E%3CP%3EThanks%20%3CA%20href%3D%22https%3A%2F%2Ftechcommunity.microsoft.com%2Ft5%2Fuser%2Fviewprofilepage%2Fuser-id%2F1148462%22%20target%3D%22_blank%22%3E%40balajisankaran%3C%2FA%3E%26nbsp%3B%2C%20so%20the%20documentation%20is%20not%20up%20to%20date%3F%3C%2FP%3E%3CP%3E%26nbsp%3B%3C%2FP%3E%3CP%3EOne%20more%20question%3A%20do%20Data%20Flows%20(as%20activity%20within%20Synapse%20Pipelines)%20also%20benefit%20from%20the%20improvements%20and%20will%20run%20on%20a%20Spark%203.1%20cluster%20instead%20of%202.4%3F%3C%2FP%3E%3C%2FLINGO-BODY%3E
Co-Authors
Version history
Last update:
‎Sep 24 2021 09:30 AM
Updated by: