Forum Discussion

athuljc's avatar
athuljc
Copper Contributor
Sep 13, 2023

Error Installing Pandas Library into Spark pool

Hi Team, 

 

I need a bit of help understanding an issue we are facing while trying to install a pandas library into a spark pool cluster that we have on synapse.

 

Steps followed,

 

1. Create a new requirements.txt file with the following text

 

also tried below

 

Step 2: Upload this requirements.txt file to the sparkpool under Packages and restart cluster. 

 

Step 3: Post successful application of settings, run a notebook under the cluster.

 

 

 

Notebook returns the below error, 

 

 

Basically this change makes the spark pool unusable.

 

Would you be able to provide any insights on why this keeps happening ?

 

Regards,
Athul

  • adityansnair's avatar
    adityansnair
    Copper Contributor
    Try creating a spark v3.2 runtime spark pool and run the import in a notebook with the new spark pool. Let me know how it goes.

Resources