spark pool taking more than 10m mins to start

Occasional Visitor

I have created 3 notebook by selecting the language as Pyspark.

spark pool is taking more then 10 minutes when starts executing the notebook.

Running these notebooks sequentially, each time spark pool starts and stops. Although the setting to pause the Spark pool is set to 60 minutes.

 

Please let me know is there any workaround to fast the start of spark pool, And same spark pool can be used in all the 3 notebooks without stopping.

3 Replies

@jasleen13  did you get answer  ?

I am wondering the same thing that @kashif_mahmood is.  I know Azure Synapse Spark Pools does not work like Databricks does, but this can get painful real quick.

i don't have a direct answer for this, because we used databricks for spark. Can you create a databricks cluster, and invoke the job from Synapse pipeline and check ? Because, in databricks you can utilize "cluster pool" concept. Check and lets us know.