In Azure Synapse, system configurations of spark pool look like below, where the number of executors, vcores, memory is defined by default.
There could be the requirement of few users who want to manipulate the number of executors or memory assigned to a spark session during execution time.
Usually, we can reconfigure them by traversing to the Spark pool on Azure Portal and set the configurations in the spark pool by uploading text file which looks like this:
But in the Synapse spark pool, few of these user-defined configurations get overridden by the default value of the Spark pool.
What should be the next step to persist these configurations at the spark pool Session level?
If we want to set config of a session with more than the executors defined at the system level (in this case there are 2 executors as we saw above), we need to write below sample code to populate the session with 4 executors. This sample code helps to logically get more executors for a session.
Execute the below code to confirm that the number of executors is the same as defined in the session which is 4 :
In the sparkUI you can also see these executors if you want to cross verify :