Forum Discussion
athuljc
Sep 13, 2023Copper Contributor
Error Installing Pandas Library into Spark pool
Hi Team,
I need a bit of help understanding an issue we are facing while trying to install a pandas library into a spark pool cluster that we have on synapse.
Steps followed,
1. Create a new requirements.txt file with the following text
also tried below
Step 2: Upload this requirements.txt file to the sparkpool under Packages and restart cluster.
Step 3: Post successful application of settings, run a notebook under the cluster.
Notebook returns the below error,
Basically this change makes the spark pool unusable.
Would you be able to provide any insights on why this keeps happening ?
Regards,
Athul
- adityansnairCopper ContributorTry creating a spark v3.2 runtime spark pool and run the import in a notebook with the new spark pool. Let me know how it goes.