Error Installing Pandas Library into Spark pool

Copper Contributor

Hi Team, 

 

I need a bit of help understanding an issue we are facing while trying to install a pandas library into a spark pool cluster that we have on synapse.

 

Steps followed,

 

1. Create a new requirements.txt file with the following text

athuljc_0-1694599836121.png

 

also tried below

athuljc_1-1694599873988.png

 

Step 2: Upload this requirements.txt file to the sparkpool under Packages and restart cluster. 

 

Step 3: Post successful application of settings, run a notebook under the cluster.

 

athuljc_2-1694599996968.png

 

 

Notebook returns the below error, 

 

athuljc_3-1694600253888.png

 

Basically this change makes the spark pool unusable.

 

Would you be able to provide any insights on why this keeps happening ?

 

Regards,
Athul

1 Reply
Try creating a spark v3.2 runtime spark pool and run the import in a notebook with the new spark pool. Let me know how it goes.