How to set Spark / Pyspark custom configs in Synapse Workspace spark pool

Published 02-05-2021 06:51 AM 1,908 Views
Microsoft

In Azure Synapse, system configurations of spark pool look like below, where the number of executors, vcores, memory is defined by default.

mubhashk_0-1612465171017.png

 

There could be the requirement of few users who want to manipulate the number of executors or memory assigned to a spark session during execution time.

 

Usually, we can reconfigure them by traversing to the Spark pool on Azure Portal and set the configurations in the spark pool by uploading text file which looks like this:

 

mubhashk_1-1612465200341.png

 

mubhashk_2-1612465230427.png

 

But in the Synapse spark pool, few of these user-defined configurations get overridden by the default value of the Spark pool.

 

What should be the next step to persist these configurations at the spark pool Session level?

 

For notebooks

If we want to set config of a session with more than the executors defined at the system level (in this case there are 2 executors as we saw above), we need to write below sample code to populate the session with 4 executors. This sample code helps to logically get more executors for a session.

mubhashk_3-1612465255185.png

Execute the below code to confirm that the number of executors is the same as defined in the session which is 4 :

mubhashk_4-1612465272088.png

In the sparkUI you can also see these executors if you want to cross verify :

mubhashk_5-1612465293726.png

A list of many session configs is briefed here 

 

We can also setup the desired session-level configuration in Apache Spark Job definition :

 

For Apache Spark Job:

 

If we want to add those configurations to our job, we have to set them when we initialize the Spark session or Spark context, for example for a PySpark job:

 

Spark Session:

 

from pyspark.sql import SparkSession

 

if __name__ == "__main__":

     

      # create Spark session with necessary configuration

      spark = SparkSession \

    .builder \

    .appName("testApp") \

    .config("spark.executor.instances","4") \

    .config("spark.executor.cores","4") \

    .getOrCreate()

 

Spark Context:

 

from pyspark import SparkContext, SparkConf

 

if __name__ == "__main__":

     

      # create Spark context with necessary configuration

      conf = SparkConf().setAppName("testApp").set("spark.hadoop.validateOutputSpecs", "false").set("spark.executor.cores","4").set("spark.executor.instances","4")

      spark = SparkContext(conf=conf)

 

 

Hope this helps you to configure a job/notebook as per your convenience with the number of executors.

 

 

%3CLINGO-SUB%20id%3D%22lingo-sub-2114434%22%20slang%3D%22en-US%22%3EHow%20to%20set%20Spark%20%2F%20Pyspark%20custom%20configs%20in%20Synapse%20Workspace%20spark%20pool%3C%2FLINGO-SUB%3E%3CLINGO-BODY%20id%3D%22lingo-body-2114434%22%20slang%3D%22en-US%22%3E%3CP%3EIn%20Azure%20Synapse%2C%20system%20configurations%20of%20spark%20pool%20look%20like%20below%2C%20where%20the%20number%20of%20executors%2C%20vcores%2C%20memory%20is%20defined%20by%20default.%3C%2FP%3E%0A%3CP%3E%3CSPAN%20class%3D%22lia-inline-image-display-wrapper%20lia-image-align-inline%22%20image-alt%3D%22mubhashk_0-1612465171017.png%22%20style%3D%22width%3A%20400px%3B%22%3E%3CIMG%20src%3D%22https%3A%2F%2Ftechcommunity.microsoft.com%2Ft5%2Fimage%2Fserverpage%2Fimage-id%2F252377i412091C1D868544F%2Fimage-size%2Fmedium%3Fv%3D1.0%26amp%3Bpx%3D400%22%20role%3D%22button%22%20title%3D%22mubhashk_0-1612465171017.png%22%20alt%3D%22mubhashk_0-1612465171017.png%22%20%2F%3E%3C%2FSPAN%3E%3C%2FP%3E%0A%3CP%3E%26nbsp%3B%3C%2FP%3E%0A%3CP%3EThere%20could%20be%20the%20requirement%20of%20few%20users%20who%20want%20to%20manipulate%20the%20number%20of%20executors%20or%20memory%20assigned%20to%20a%20spark%20session%20during%20execution%20time.%3C%2FP%3E%0A%3CP%3E%26nbsp%3B%3C%2FP%3E%0A%3CP%3EUsually%2C%20we%20can%20reconfigure%20them%20by%20traversing%20to%20the%20Spark%20pool%20on%20Azure%20Portal%20and%20set%20the%20configurations%20in%20the%20spark%20pool%20by%20uploading%20text%20file%20which%20looks%20like%20this%3A%3C%2FP%3E%0A%3CP%3E%26nbsp%3B%3C%2FP%3E%0A%3CP%3E%3CSPAN%20class%3D%22lia-inline-image-display-wrapper%20lia-image-align-inline%22%20image-alt%3D%22mubhashk_1-1612465200341.png%22%20style%3D%22width%3A%20400px%3B%22%3E%3CIMG%20src%3D%22https%3A%2F%2Ftechcommunity.microsoft.com%2Ft5%2Fimage%2Fserverpage%2Fimage-id%2F252380i5C295CB4D8AB7900%2Fimage-size%2Fmedium%3Fv%3D1.0%26amp%3Bpx%3D400%22%20role%3D%22button%22%20title%3D%22mubhashk_1-1612465200341.png%22%20alt%3D%22mubhashk_1-1612465200341.png%22%20%2F%3E%3C%2FSPAN%3E%3C%2FP%3E%0A%3CP%3E%26nbsp%3B%3C%2FP%3E%0A%3CP%3E%3CSPAN%20class%3D%22lia-inline-image-display-wrapper%20lia-image-align-inline%22%20image-alt%3D%22mubhashk_2-1612465230427.png%22%20style%3D%22width%3A%20400px%3B%22%3E%3CIMG%20src%3D%22https%3A%2F%2Ftechcommunity.microsoft.com%2Ft5%2Fimage%2Fserverpage%2Fimage-id%2F252381i0446957F93ACF486%2Fimage-size%2Fmedium%3Fv%3D1.0%26amp%3Bpx%3D400%22%20role%3D%22button%22%20title%3D%22mubhashk_2-1612465230427.png%22%20alt%3D%22mubhashk_2-1612465230427.png%22%20%2F%3E%3C%2FSPAN%3E%3C%2FP%3E%0A%3CP%3E%26nbsp%3B%3C%2FP%3E%0A%3CP%3EBut%20in%20the%20Synapse%20spark%20pool%2C%20few%20of%20these%20user-defined%20configurations%20get%20overridden%20by%20the%20default%20value%20of%20the%20Spark%20pool.%3C%2FP%3E%0A%3CP%3E%26nbsp%3B%3C%2FP%3E%0A%3CP%3EWhat%20should%20be%20the%20next%20step%20to%20persist%20these%20configurations%20at%20the%20spark%20pool%20Session%20level%3F%3C%2FP%3E%0A%3CP%3E%26nbsp%3B%3C%2FP%3E%0A%3CP%3E%3CSTRONG%3E%3CU%3EFor%20notebooks%3C%2FU%3E%3C%2FSTRONG%3E%3C%2FP%3E%0A%3CP%3EIf%20we%20want%20to%20set%20config%20of%20a%20session%20with%20more%20than%20the%20executors%20defined%20at%20the%20system%20level%20(in%20this%20case%20there%20are%202%20executors%20as%20we%20saw%20above)%2C%20we%20need%20to%20write%20below%20sample%20code%20to%20populate%20the%20session%20with%204%20executors.%20This%20sample%20code%20helps%20to%20logically%20get%20more%20executors%20for%20a%20session.%3C%2FP%3E%0A%3CP%3E%3CSPAN%20class%3D%22lia-inline-image-display-wrapper%20lia-image-align-inline%22%20image-alt%3D%22mubhashk_3-1612465255185.png%22%20style%3D%22width%3A%20400px%3B%22%3E%3CIMG%20src%3D%22https%3A%2F%2Ftechcommunity.microsoft.com%2Ft5%2Fimage%2Fserverpage%2Fimage-id%2F252382iA35D3790A70E250B%2Fimage-size%2Fmedium%3Fv%3D1.0%26amp%3Bpx%3D400%22%20role%3D%22button%22%20title%3D%22mubhashk_3-1612465255185.png%22%20alt%3D%22mubhashk_3-1612465255185.png%22%20%2F%3E%3C%2FSPAN%3E%3C%2FP%3E%0A%3CP%3EExecute%20the%20below%20code%20to%20confirm%20that%20the%20number%20of%20executors%20is%20the%20same%20as%20defined%20in%20the%20session%20which%20is%204%20%3A%3C%2FP%3E%0A%3CP%3E%3CSPAN%20class%3D%22lia-inline-image-display-wrapper%20lia-image-align-inline%22%20image-alt%3D%22mubhashk_4-1612465272088.png%22%20style%3D%22width%3A%20400px%3B%22%3E%3CIMG%20src%3D%22https%3A%2F%2Ftechcommunity.microsoft.com%2Ft5%2Fimage%2Fserverpage%2Fimage-id%2F252383iEEDB12ED3DB42504%2Fimage-size%2Fmedium%3Fv%3D1.0%26amp%3Bpx%3D400%22%20role%3D%22button%22%20title%3D%22mubhashk_4-1612465272088.png%22%20alt%3D%22mubhashk_4-1612465272088.png%22%20%2F%3E%3C%2FSPAN%3E%3C%2FP%3E%0A%3CP%3EIn%20the%20sparkUI%20you%20can%20also%20see%20these%20executors%20if%20you%20want%20to%20cross%20verify%20%3A%3C%2FP%3E%0A%3CP%3E%3CSPAN%20class%3D%22lia-inline-image-display-wrapper%20lia-image-align-inline%22%20image-alt%3D%22mubhashk_5-1612465293726.png%22%20style%3D%22width%3A%20400px%3B%22%3E%3CIMG%20src%3D%22https%3A%2F%2Ftechcommunity.microsoft.com%2Ft5%2Fimage%2Fserverpage%2Fimage-id%2F252384iA3E5AD67DFE86938%2Fimage-size%2Fmedium%3Fv%3D1.0%26amp%3Bpx%3D400%22%20role%3D%22button%22%20title%3D%22mubhashk_5-1612465293726.png%22%20alt%3D%22mubhashk_5-1612465293726.png%22%20%2F%3E%3C%2FSPAN%3E%3C%2FP%3E%0A%3CP%3EA%20list%20of%20many%20session%20configs%20is%20briefed%26nbsp%3B%3CA%20href%3D%22https%3A%2F%2Fgithub.com%2Fcloudera%2Flivy%23request-body%22%20target%3D%22_blank%22%20rel%3D%22noopener%20noreferrer%22%3Ehere%26nbsp%3B%3C%2FA%3E.%26nbsp%3B%3C%2FP%3E%0A%3CP%3E%26nbsp%3B%3C%2FP%3E%0A%3CP%3EWe%20can%20also%20setup%20the%20desired%20session-level%20configuration%20in%20Apache%20Spark%20Job%20definition%20%3A%3C%2FP%3E%0A%3CP%3E%26nbsp%3B%3C%2FP%3E%0A%3CP%3E%3CSTRONG%3E%3CU%3EFor%20Apache%20Spark%20Job%3A%3C%2FU%3E%3C%2FSTRONG%3E%3C%2FP%3E%0A%3CP%3E%26nbsp%3B%3C%2FP%3E%0A%3CP%3EIf%20we%20want%20to%20add%20those%20configurations%20to%20our%20job%2C%20we%20have%20to%20set%20them%20when%20we%20initialize%20the%20Spark%20session%20or%20Spark%20context%2C%20for%20example%20for%20a%20PySpark%20job%3A%3C%2FP%3E%0A%3CP%3E%26nbsp%3B%3C%2FP%3E%0A%3CP%3ESpark%20Session%3A%3C%2FP%3E%0A%3CP%3E%26nbsp%3B%3C%2FP%3E%0A%3CP%3Efrom%20pyspark.sql%20import%20SparkSession%3C%2FP%3E%0A%3CP%3E%26nbsp%3B%3C%2FP%3E%0A%3CP%3Eif%20__name__%20%3D%3D%20%22__main__%22%3A%3C%2FP%3E%0A%3CP%3E%26nbsp%3B%26nbsp%3B%26nbsp%3B%26nbsp%3B%26nbsp%3B%3C%2FP%3E%0A%3CP%3E%26nbsp%3B%26nbsp%3B%26nbsp%3B%26nbsp%3B%26nbsp%3B%20%23%20create%20Spark%20session%20with%20necessary%20configuration%3C%2FP%3E%0A%3CP%3E%26nbsp%3B%26nbsp%3B%26nbsp%3B%26nbsp%3B%26nbsp%3B%20spark%20%3D%20SparkSession%20%5C%3C%2FP%3E%0A%3CP%3E%26nbsp%3B%26nbsp%3B%26nbsp%3B%20.builder%20%5C%3C%2FP%3E%0A%3CP%3E%26nbsp%3B%26nbsp%3B%26nbsp%3B%20.appName(%22testApp%22)%20%5C%3C%2FP%3E%0A%3CP%3E%26nbsp%3B%26nbsp%3B%26nbsp%3B%26nbsp%3B.config(%22spark.executor.instances%22%2C%224%22)%20%5C%3C%2FP%3E%0A%3CP%3E%26nbsp%3B%26nbsp%3B%26nbsp%3B%26nbsp%3B.config(%22spark.executor.cores%22%2C%224%22)%20%5C%3C%2FP%3E%0A%3CP%3E%26nbsp%3B%26nbsp%3B%26nbsp%3B%20.getOrCreate()%3C%2FP%3E%0A%3CP%3E%26nbsp%3B%3C%2FP%3E%0A%3CP%3ESpark%20Context%3A%3C%2FP%3E%0A%3CP%3E%26nbsp%3B%3C%2FP%3E%0A%3CP%3Efrom%20pyspark%20import%20SparkContext%2C%20SparkConf%3C%2FP%3E%0A%3CP%3E%26nbsp%3B%3C%2FP%3E%0A%3CP%3Eif%20__name__%20%3D%3D%20%22__main__%22%3A%3C%2FP%3E%0A%3CP%3E%26nbsp%3B%26nbsp%3B%26nbsp%3B%26nbsp%3B%26nbsp%3B%3C%2FP%3E%0A%3CP%3E%26nbsp%3B%26nbsp%3B%26nbsp%3B%26nbsp%3B%26nbsp%3B%20%23%20create%20Spark%20context%20with%20necessary%20configuration%3C%2FP%3E%0A%3CP%3E%26nbsp%3B%26nbsp%3B%26nbsp%3B%26nbsp%3B%26nbsp%3B%20conf%20%3D%20SparkConf().setAppName(%22testApp%22).set(%22spark.hadoop.validateOutputSpecs%22%2C%20%22false%22).set(%22spark.executor.cores%22%2C%224%22).set(%22spark.executor.instances%22%2C%224%22)%3C%2FP%3E%0A%3CP%3E%26nbsp%3B%26nbsp%3B%26nbsp%3B%26nbsp%3B%26nbsp%3B%20spark%20%3D%20SparkContext(conf%3Dconf)%3C%2FP%3E%0A%3CP%3E%26nbsp%3B%3C%2FP%3E%0A%3CP%3E%26nbsp%3B%3C%2FP%3E%0A%3CP%3EHope%20this%20helps%20you%20to%20configure%20a%20job%2Fnotebook%20as%20per%20your%20convenience%20with%20the%20number%20of%20executors.%3C%2FP%3E%0A%3CP%3E%26nbsp%3B%3C%2FP%3E%0A%3CP%3E%26nbsp%3B%3C%2FP%3E%3C%2FLINGO-BODY%3E%3CLINGO-LABS%20id%3D%22lingo-labs-2114434%22%20slang%3D%22en-US%22%3E%3CLINGO-LABEL%3ESynapse%20DevOps%3C%2FLINGO-LABEL%3E%3CLINGO-LABEL%3ESynapse%20Spark%3C%2FLINGO-LABEL%3E%3C%2FLINGO-LABS%3E
Co-Authors
Version history
Last update:
‎Feb 05 2021 01:46 AM
Updated by: